From python-checkins at python.org Thu Jan 1 01:12:48 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 01 Jan 2015 00:12:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_update_for_cop?= =?utf-8?q?yright_for_2015?= Message-ID: <20150101001232.11567.23068@psf.io> https://hg.python.org/cpython/rev/7d97f4d5fd36 changeset: 93999:7d97f4d5fd36 branch: 2.7 parent: 93984:8f92ab37dd3a user: Benjamin Peterson date: Wed Dec 31 18:09:36 2014 -0600 summary: update for copyright for 2015 files: Doc/copyright.rst | 2 +- Doc/license.rst | 2 +- LICENSE | 4 ++-- PC/python_nt.rc | 2 +- Python/getcopyright.c | 2 +- README | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Doc/copyright.rst b/Doc/copyright.rst --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Python and this documentation is: -Copyright ?? 2001-2014 Python Software Foundation. All rights reserved. +Copyright ?? 2001-2015 Python Software Foundation. All rights reserved. Copyright ?? 2000 BeOpen.com. All rights reserved. diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -84,7 +84,7 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright ?? 2001-2014 Python Software Foundation; All Rights + copyright, i.e., "Copyright ?? 2001-2015 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -74,8 +74,8 @@ distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained -in Python alone or in any derivative version prepared by Licensee. +2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" +are retained in Python alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make diff --git a/PC/python_nt.rc b/PC/python_nt.rc --- a/PC/python_nt.rc +++ b/PC/python_nt.rc @@ -61,7 +61,7 @@ VALUE "FileDescription", "Python Core\0" VALUE "FileVersion", PYTHON_VERSION VALUE "InternalName", "Python DLL\0" - VALUE "LegalCopyright", "Copyright ? 2001-2014 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" + VALUE "LegalCopyright", "Copyright ? 2001-2015 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" VALUE "OriginalFilename", PYTHON_DLL_NAME "\0" VALUE "ProductName", "Python\0" VALUE "ProductVersion", PYTHON_VERSION diff --git a/Python/getcopyright.c b/Python/getcopyright.c --- a/Python/getcopyright.c +++ b/Python/getcopyright.c @@ -4,7 +4,7 @@ static char cprt[] = "\ -Copyright (c) 2001-2014 Python Software Foundation.\n\ +Copyright (c) 2001-2015 Python Software Foundation.\n\ All Rights Reserved.\n\ \n\ Copyright (c) 2000 BeOpen.com.\n\ diff --git a/README b/README --- a/README +++ b/README @@ -2,7 +2,7 @@ ============================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -2012, 2013, 2014 Python Software Foundation. All rights reserved. +2012, 2013, 2014, 2015 Python Software Foundation. All rights reserved. Copyright (c) 2000 BeOpen.com. All rights reserved. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 1 01:12:49 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 01 Jan 2015 00:12:49 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_update_for_cop?= =?utf-8?q?yright_for_2015?= Message-ID: <20150101001231.72575.63690@psf.io> https://hg.python.org/cpython/rev/1806a00fd6b8 changeset: 93995:1806a00fd6b8 branch: 3.2 parent: 93985:223d0927e27d user: Benjamin Peterson date: Wed Dec 31 18:09:36 2014 -0600 summary: update for copyright for 2015 files: Doc/copyright.rst | 2 +- Doc/license.rst | 2 +- LICENSE | 4 ++-- PC/python_nt.rc | 2 +- Python/getcopyright.c | 2 +- README | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Doc/copyright.rst b/Doc/copyright.rst --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Python and this documentation is: -Copyright ?? 2001-2013 Python Software Foundation. All rights reserved. +Copyright ?? 2001-2015 Python Software Foundation. All rights reserved. Copyright ?? 2000 BeOpen.com. All rights reserved. diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -154,7 +154,7 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright ?? 2001-2014 Python Software Foundation; All Rights + copyright, i.e., "Copyright ?? 2001-2015 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -112,8 +112,8 @@ distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained -in Python alone or in any derivative version prepared by Licensee. +2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" +are retained in Python alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make diff --git a/PC/python_nt.rc b/PC/python_nt.rc --- a/PC/python_nt.rc +++ b/PC/python_nt.rc @@ -61,7 +61,7 @@ VALUE "FileDescription", "Python Core\0" VALUE "FileVersion", PYTHON_VERSION VALUE "InternalName", "Python DLL\0" - VALUE "LegalCopyright", "Copyright ? 2001-2014 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" + VALUE "LegalCopyright", "Copyright ? 2001-2015 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" VALUE "OriginalFilename", PYTHON_DLL_NAME "\0" VALUE "ProductName", "Python\0" VALUE "ProductVersion", PYTHON_VERSION diff --git a/Python/getcopyright.c b/Python/getcopyright.c --- a/Python/getcopyright.c +++ b/Python/getcopyright.c @@ -4,7 +4,7 @@ static char cprt[] = "\ -Copyright (c) 2001-2014 Python Software Foundation.\n\ +Copyright (c) 2001-2015 Python Software Foundation.\n\ All Rights Reserved.\n\ \n\ Copyright (c) 2000 BeOpen.com.\n\ diff --git a/README b/README --- a/README +++ b/README @@ -2,7 +2,7 @@ ============================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -2012, 2013, 2014 Python Software Foundation. All rights reserved. +2012, 2013, 2014, 2015 Python Software Foundation. All rights reserved. Python 3.x is a new version of the language, which is incompatible with the 2.x line of releases. The language is mostly the same, but many details, especially -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 1 01:12:49 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 01 Jan 2015 00:12:49 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_merge_3=2E3?= Message-ID: <20150101001232.72555.99543@psf.io> https://hg.python.org/cpython/rev/9c6bbd4fea86 changeset: 93997:9c6bbd4fea86 branch: 3.4 parent: 93993:29689050ec78 parent: 93996:3b202cc79a38 user: Benjamin Peterson date: Wed Dec 31 18:11:22 2014 -0600 summary: merge 3.3 files: Doc/copyright.rst | 2 +- Doc/license.rst | 2 +- LICENSE | 4 ++-- PC/python_nt.rc | 2 +- Python/getcopyright.c | 2 +- README | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Doc/copyright.rst b/Doc/copyright.rst --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Python and this documentation is: -Copyright ?? 2001-2014 Python Software Foundation. All rights reserved. +Copyright ?? 2001-2015 Python Software Foundation. All rights reserved. Copyright ?? 2000 BeOpen.com. All rights reserved. diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -84,7 +84,7 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright ?? 2001-2014 Python Software Foundation; All + copyright, i.e., "Copyright ?? 2001-2015 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -74,8 +74,8 @@ distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are -retained in Python alone or in any derivative version prepared by Licensee. +2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" +are retained in Python alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make diff --git a/PC/python_nt.rc b/PC/python_nt.rc --- a/PC/python_nt.rc +++ b/PC/python_nt.rc @@ -61,7 +61,7 @@ VALUE "FileDescription", "Python Core\0" VALUE "FileVersion", PYTHON_VERSION VALUE "InternalName", "Python DLL\0" - VALUE "LegalCopyright", "Copyright ? 2001-2014 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" + VALUE "LegalCopyright", "Copyright ? 2001-2015 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" VALUE "OriginalFilename", PYTHON_DLL_NAME "\0" VALUE "ProductName", "Python\0" VALUE "ProductVersion", PYTHON_VERSION diff --git a/Python/getcopyright.c b/Python/getcopyright.c --- a/Python/getcopyright.c +++ b/Python/getcopyright.c @@ -4,7 +4,7 @@ static const char cprt[] = "\ -Copyright (c) 2001-2014 Python Software Foundation.\n\ +Copyright (c) 2001-2015 Python Software Foundation.\n\ All Rights Reserved.\n\ \n\ Copyright (c) 2000 BeOpen.com.\n\ diff --git a/README b/README --- a/README +++ b/README @@ -2,7 +2,7 @@ ============================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -2012, 2013, 2014 Python Software Foundation. All rights reserved. +2012, 2013, 2014, 2015 Python Software Foundation. All rights reserved. Python 3.x is a new version of the language, which is incompatible with the 2.x line of releases. The language is mostly the same, but many details, especially -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 1 01:12:48 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 01 Jan 2015 00:12:48 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E2?= Message-ID: <20150101001232.72581.27067@psf.io> https://hg.python.org/cpython/rev/3b202cc79a38 changeset: 93996:3b202cc79a38 branch: 3.3 parent: 93986:e15d93926e47 parent: 93995:1806a00fd6b8 user: Benjamin Peterson date: Wed Dec 31 18:10:13 2014 -0600 summary: merge 3.2 files: Doc/copyright.rst | 2 +- Doc/license.rst | 2 +- LICENSE | 4 ++-- PC/python_nt.rc | 2 +- Python/getcopyright.c | 2 +- README | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Doc/copyright.rst b/Doc/copyright.rst --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Python and this documentation is: -Copyright ?? 2001-2014 Python Software Foundation. All rights reserved. +Copyright ?? 2001-2015 Python Software Foundation. All rights reserved. Copyright ?? 2000 BeOpen.com. All rights reserved. diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -84,7 +84,7 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright ?? 2001-2014 Python Software Foundation; All Rights + copyright, i.e., "Copyright ?? 2001-2015 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -74,8 +74,8 @@ distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained -in Python alone or in any derivative version prepared by Licensee. +2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" +are retained in Python alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make diff --git a/PC/python_nt.rc b/PC/python_nt.rc --- a/PC/python_nt.rc +++ b/PC/python_nt.rc @@ -61,7 +61,7 @@ VALUE "FileDescription", "Python Core\0" VALUE "FileVersion", PYTHON_VERSION VALUE "InternalName", "Python DLL\0" - VALUE "LegalCopyright", "Copyright ? 2001-2014 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" + VALUE "LegalCopyright", "Copyright ? 2001-2015 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" VALUE "OriginalFilename", PYTHON_DLL_NAME "\0" VALUE "ProductName", "Python\0" VALUE "ProductVersion", PYTHON_VERSION diff --git a/Python/getcopyright.c b/Python/getcopyright.c --- a/Python/getcopyright.c +++ b/Python/getcopyright.c @@ -4,7 +4,7 @@ static const char cprt[] = "\ -Copyright (c) 2001-2014 Python Software Foundation.\n\ +Copyright (c) 2001-2015 Python Software Foundation.\n\ All Rights Reserved.\n\ \n\ Copyright (c) 2000 BeOpen.com.\n\ diff --git a/README b/README --- a/README +++ b/README @@ -2,7 +2,7 @@ ============================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -2012, 2013, 2014 Python Software Foundation. All rights reserved. +2012, 2013, 2014, 2015 Python Software Foundation. All rights reserved. Python 3.x is a new version of the language, which is incompatible with the 2.x line of releases. The language is mostly the same, but many details, especially -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 1 01:12:48 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 01 Jan 2015 00:12:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <20150101001232.125892.62097@psf.io> https://hg.python.org/cpython/rev/3dba1fed51a0 changeset: 93998:3dba1fed51a0 parent: 93994:01437956ea67 parent: 93997:9c6bbd4fea86 user: Benjamin Peterson date: Wed Dec 31 18:11:34 2014 -0600 summary: merge 3.4 files: Doc/copyright.rst | 2 +- Doc/license.rst | 2 +- LICENSE | 4 ++-- PC/python_nt.rc | 2 +- Python/getcopyright.c | 2 +- README | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Doc/copyright.rst b/Doc/copyright.rst --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Python and this documentation is: -Copyright ?? 2001-2014 Python Software Foundation. All rights reserved. +Copyright ?? 2001-2015 Python Software Foundation. All rights reserved. Copyright ?? 2000 BeOpen.com. All rights reserved. diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -84,7 +84,7 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright ?? 2001-2014 Python Software Foundation; All + copyright, i.e., "Copyright ?? 2001-2015 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -74,8 +74,8 @@ distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are -retained in Python alone or in any derivative version prepared by Licensee. +2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" +are retained in Python alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make diff --git a/PC/python_nt.rc b/PC/python_nt.rc --- a/PC/python_nt.rc +++ b/PC/python_nt.rc @@ -61,7 +61,7 @@ VALUE "FileDescription", "Python Core\0" VALUE "FileVersion", PYTHON_VERSION VALUE "InternalName", "Python DLL\0" - VALUE "LegalCopyright", "Copyright ? 2001-2014 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" + VALUE "LegalCopyright", "Copyright ? 2001-2015 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" VALUE "OriginalFilename", PYTHON_DLL_NAME "\0" VALUE "ProductName", "Python\0" VALUE "ProductVersion", PYTHON_VERSION diff --git a/Python/getcopyright.c b/Python/getcopyright.c --- a/Python/getcopyright.c +++ b/Python/getcopyright.c @@ -4,7 +4,7 @@ static const char cprt[] = "\ -Copyright (c) 2001-2014 Python Software Foundation.\n\ +Copyright (c) 2001-2015 Python Software Foundation.\n\ All Rights Reserved.\n\ \n\ Copyright (c) 2000 BeOpen.com.\n\ diff --git a/README b/README --- a/README +++ b/README @@ -2,7 +2,7 @@ ==================================== Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -2012, 2013, 2014 Python Software Foundation. All rights reserved. +2012, 2013, 2014, 2015 Python Software Foundation. All rights reserved. Python 3.x is a new version of the language, which is incompatible with the 2.x line of releases. The language is mostly the same, but many details, especially -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 1 01:31:34 2015 From: python-checkins at python.org (ned.deily) Date: Thu, 01 Jan 2015 00:31:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Update_copyrig?= =?utf-8?q?ht_dates_in_OS_X_installer=2E?= Message-ID: <20150101003131.8749.5306@psf.io> https://hg.python.org/cpython/rev/4e067cf7e299 changeset: 94000:4e067cf7e299 branch: 2.7 user: Ned Deily date: Wed Dec 31 16:30:09 2014 -0800 summary: Update copyright dates in OS X installer. files: Mac/BuildScript/resources/License.rtf | 14 +++++--------- Mac/IDLE/Info.plist.in | 2 +- Mac/PythonLauncher/Info.plist.in | 2 +- Mac/Resources/app/Info.plist.in | 6 +++--- Mac/Resources/framework/Info.plist.in | 4 ++-- 5 files changed, 12 insertions(+), 16 deletions(-) diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf --- a/Mac/BuildScript/resources/License.rtf +++ b/Mac/BuildScript/resources/License.rtf @@ -54,7 +54,7 @@ \b0 \ 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using this software ("Python") in source or binary form and its associated documentation.\ \ -2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\ +2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\ \ 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python.\ \ @@ -124,22 +124,18 @@ STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.\ \ \ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\b \cf0 \ul \ulc0 LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural +\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\ -\b0 \cf0 \ulnone \ +\b0 \ulnone \ This installer incorporates portions of the following third-party software:\ \ \f2 $THIRD_PARTY_LIBS\ \ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\f0 \cf0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\cf0 \ +\f0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\ +\ \ \ \ diff --git a/Mac/IDLE/Info.plist.in b/Mac/IDLE/Info.plist.in --- a/Mac/IDLE/Info.plist.in +++ b/Mac/IDLE/Info.plist.in @@ -36,7 +36,7 @@ CFBundleExecutable IDLE CFBundleGetInfoString - %VERSION%, ? 2001-2014 Python Software Foundation + %VERSION%, ? 2001-2015 Python Software Foundation CFBundleIconFile IDLE.icns CFBundleIdentifier diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in --- a/Mac/PythonLauncher/Info.plist.in +++ b/Mac/PythonLauncher/Info.plist.in @@ -40,7 +40,7 @@ CFBundleExecutable PythonLauncher CFBundleGetInfoString - %VERSION%, ? 2001-2014 Python Software Foundation + %VERSION%, ? 2001-2015 Python Software Foundation CFBundleIconFile PythonLauncher.icns CFBundleIdentifier diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in --- a/Mac/Resources/app/Info.plist.in +++ b/Mac/Resources/app/Info.plist.in @@ -20,7 +20,7 @@ CFBundleExecutable Python CFBundleGetInfoString - %version%, (c) 2004-2014 Python Software Foundation. + %version%, (c) 2001-2015 Python Software Foundation. CFBundleHelpBookFolder Documentation @@ -37,7 +37,7 @@ CFBundleInfoDictionaryVersion 6.0 CFBundleLongVersionString - %version%, (c) 2004-2014 Python Software Foundation. + %version%, (c) 2001-2015 Python Software Foundation. CFBundleName Python CFBundlePackageType @@ -55,7 +55,7 @@ NSAppleScriptEnabled NSHumanReadableCopyright - (c) 2014 Python Software Foundation. + (c) 2001-2015 Python Software Foundation. NSHighResolutionCapable diff --git a/Mac/Resources/framework/Info.plist.in b/Mac/Resources/framework/Info.plist.in --- a/Mac/Resources/framework/Info.plist.in +++ b/Mac/Resources/framework/Info.plist.in @@ -17,9 +17,9 @@ CFBundlePackageType FMWK CFBundleShortVersionString - %VERSION%, (c) 2004-2014 Python Software Foundation. + %VERSION%, (c) 2001-2015 Python Software Foundation. CFBundleLongVersionString - %VERSION%, (c) 2004-2014 Python Software Foundation. + %VERSION%, (c) 2001-2015 Python Software Foundation. CFBundleSignature ???? CFBundleVersion -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 1 01:31:34 2015 From: python-checkins at python.org (ned.deily) Date: Thu, 01 Jan 2015 00:31:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Update_copyrig?= =?utf-8?q?ht_dates_in_OS_X_installer=2E?= Message-ID: <20150101003131.8743.91385@psf.io> https://hg.python.org/cpython/rev/50d581f69a73 changeset: 94001:50d581f69a73 branch: 3.4 parent: 93997:9c6bbd4fea86 user: Ned Deily date: Wed Dec 31 16:30:26 2014 -0800 summary: Update copyright dates in OS X installer. files: Mac/BuildScript/resources/License.rtf | 14 +++++--------- Mac/IDLE/IDLE.app/Contents/Info.plist | 2 +- Mac/PythonLauncher/Info.plist.in | 2 +- Mac/Resources/app/Info.plist.in | 6 +++--- Mac/Resources/framework/Info.plist.in | 4 ++-- 5 files changed, 12 insertions(+), 16 deletions(-) diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf --- a/Mac/BuildScript/resources/License.rtf +++ b/Mac/BuildScript/resources/License.rtf @@ -54,7 +54,7 @@ \b0 \ 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using this software ("Python") in source or binary form and its associated documentation.\ \ -2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\ +2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\ \ 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python.\ \ @@ -124,22 +124,18 @@ STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.\ \ \ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\b \cf0 \ul \ulc0 LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural +\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\ -\b0 \cf0 \ulnone \ +\b0 \ulnone \ This installer incorporates portions of the following third-party software:\ \ \f2 $THIRD_PARTY_LIBS\ \ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\f0 \cf0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\cf0 \ +\f0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\ +\ \ \ \ diff --git a/Mac/IDLE/IDLE.app/Contents/Info.plist b/Mac/IDLE/IDLE.app/Contents/Info.plist --- a/Mac/IDLE/IDLE.app/Contents/Info.plist +++ b/Mac/IDLE/IDLE.app/Contents/Info.plist @@ -36,7 +36,7 @@ CFBundleExecutable IDLE CFBundleGetInfoString - %version%, ? 2001-2014 Python Software Foundation + %version%, ? 2001-2015 Python Software Foundation CFBundleIconFile IDLE.icns CFBundleIdentifier diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in --- a/Mac/PythonLauncher/Info.plist.in +++ b/Mac/PythonLauncher/Info.plist.in @@ -40,7 +40,7 @@ CFBundleExecutable PythonLauncher CFBundleGetInfoString - %VERSION%, ? 2001-2014 Python Software Foundation + %VERSION%, ? 2001-2015 Python Software Foundation CFBundleIconFile PythonLauncher.icns CFBundleIdentifier diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in --- a/Mac/Resources/app/Info.plist.in +++ b/Mac/Resources/app/Info.plist.in @@ -20,7 +20,7 @@ CFBundleExecutable Python CFBundleGetInfoString - %version%, (c) 2004-2014 Python Software Foundation. + %version%, (c) 2001-2015 Python Software Foundation. CFBundleHelpBookFolder Documentation @@ -37,7 +37,7 @@ CFBundleInfoDictionaryVersion 6.0 CFBundleLongVersionString - %version%, (c) 2004-2014 Python Software Foundation. + %version%, (c) 2001-2015 Python Software Foundation. CFBundleName Python CFBundlePackageType @@ -55,7 +55,7 @@ NSAppleScriptEnabled NSHumanReadableCopyright - (c) 2014 Python Software Foundation. + (c) 2001-2015 Python Software Foundation. NSHighResolutionCapable diff --git a/Mac/Resources/framework/Info.plist.in b/Mac/Resources/framework/Info.plist.in --- a/Mac/Resources/framework/Info.plist.in +++ b/Mac/Resources/framework/Info.plist.in @@ -17,9 +17,9 @@ CFBundlePackageType FMWK CFBundleShortVersionString - %VERSION%, (c) 2004-2014 Python Software Foundation. + %VERSION%, (c) 2001-2015 Python Software Foundation. CFBundleLongVersionString - %VERSION%, (c) 2004-2014 Python Software Foundation. + %VERSION%, (c) 2001-2015 Python Software Foundation. CFBundleSignature ???? CFBundleVersion -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 1 01:31:34 2015 From: python-checkins at python.org (ned.deily) Date: Thu, 01 Jan 2015 00:31:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Update_copyright_dates_in_OS_X_installer=2E?= Message-ID: <20150101003131.11563.90564@psf.io> https://hg.python.org/cpython/rev/4d1ae7eec0d4 changeset: 94002:4d1ae7eec0d4 parent: 93998:3dba1fed51a0 parent: 94001:50d581f69a73 user: Ned Deily date: Wed Dec 31 16:31:06 2014 -0800 summary: Update copyright dates in OS X installer. files: Mac/BuildScript/resources/License.rtf | 14 +++++--------- Mac/IDLE/IDLE.app/Contents/Info.plist | 2 +- Mac/PythonLauncher/Info.plist.in | 2 +- Mac/Resources/app/Info.plist.in | 6 +++--- Mac/Resources/framework/Info.plist.in | 4 ++-- 5 files changed, 12 insertions(+), 16 deletions(-) diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf --- a/Mac/BuildScript/resources/License.rtf +++ b/Mac/BuildScript/resources/License.rtf @@ -54,7 +54,7 @@ \b0 \ 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using this software ("Python") in source or binary form and its associated documentation.\ \ -2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\ +2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\ \ 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python.\ \ @@ -124,22 +124,18 @@ STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.\ \ \ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\b \cf0 \ul \ulc0 LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural +\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\ -\b0 \cf0 \ulnone \ +\b0 \ulnone \ This installer incorporates portions of the following third-party software:\ \ \f2 $THIRD_PARTY_LIBS\ \ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\f0 \cf0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\ -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural -\cf0 \ +\f0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\ +\ \ \ \ diff --git a/Mac/IDLE/IDLE.app/Contents/Info.plist b/Mac/IDLE/IDLE.app/Contents/Info.plist --- a/Mac/IDLE/IDLE.app/Contents/Info.plist +++ b/Mac/IDLE/IDLE.app/Contents/Info.plist @@ -36,7 +36,7 @@ CFBundleExecutable IDLE CFBundleGetInfoString - %version%, ? 2001-2014 Python Software Foundation + %version%, ? 2001-2015 Python Software Foundation CFBundleIconFile IDLE.icns CFBundleIdentifier diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in --- a/Mac/PythonLauncher/Info.plist.in +++ b/Mac/PythonLauncher/Info.plist.in @@ -40,7 +40,7 @@ CFBundleExecutable Python Launcher CFBundleGetInfoString - %VERSION%, ? 2001-2014 Python Software Foundation + %VERSION%, ? 2001-2015 Python Software Foundation CFBundleIconFile PythonLauncher.icns CFBundleIdentifier diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in --- a/Mac/Resources/app/Info.plist.in +++ b/Mac/Resources/app/Info.plist.in @@ -20,7 +20,7 @@ CFBundleExecutable Python CFBundleGetInfoString - %version%, (c) 2004-2014 Python Software Foundation. + %version%, (c) 2001-2015 Python Software Foundation. CFBundleHelpBookFolder Documentation @@ -37,7 +37,7 @@ CFBundleInfoDictionaryVersion 6.0 CFBundleLongVersionString - %version%, (c) 2004-2014 Python Software Foundation. + %version%, (c) 2001-2015 Python Software Foundation. CFBundleName Python CFBundlePackageType @@ -55,7 +55,7 @@ NSAppleScriptEnabled NSHumanReadableCopyright - (c) 2014 Python Software Foundation. + (c) 2001-2015 Python Software Foundation. NSHighResolutionCapable diff --git a/Mac/Resources/framework/Info.plist.in b/Mac/Resources/framework/Info.plist.in --- a/Mac/Resources/framework/Info.plist.in +++ b/Mac/Resources/framework/Info.plist.in @@ -17,9 +17,9 @@ CFBundlePackageType FMWK CFBundleShortVersionString - %VERSION%, (c) 2004-2014 Python Software Foundation. + %VERSION%, (c) 2001-2015 Python Software Foundation. CFBundleLongVersionString - %VERSION%, (c) 2004-2014 Python Software Foundation. + %VERSION%, (c) 2001-2015 Python Software Foundation. CFBundleSignature ???? CFBundleVersion -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Thu Jan 1 09:25:48 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 01 Jan 2015 09:25:48 +0100 Subject: [Python-checkins] Daily reference leaks (4d1ae7eec0d4): sum=3 Message-ID: results for 4d1ae7eec0d4 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog5qHHxE', '-x'] From python-checkins at python.org Thu Jan 1 14:28:35 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 01 Jan 2015 13:28:35 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323132=3A_Improve_?= =?utf-8?q?performance_and_introspection_support_of_comparison?= Message-ID: <20150101132833.72575.35086@psf.io> https://hg.python.org/cpython/rev/4e85df8b3ea6 changeset: 94003:4e85df8b3ea6 user: Serhiy Storchaka date: Thu Jan 01 15:23:12 2015 +0200 summary: Issue #23132: Improve performance and introspection support of comparison methods created by functool.total_ordering. files: Lib/functools.py | 87 ++++++++++++++----------- Lib/test/test_functools.py | 18 +++++ Misc/NEWS | 3 + 3 files changed, 69 insertions(+), 39 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -113,76 +113,85 @@ # underlying user provided method. Using this scheme, the __gt__ derived # from a user provided __lt__ becomes: # -# lambda self, other: _not_op_and_not_eq(self.__lt__, self, other)) +# 'def __gt__(self, other):' + _not_op_and_not_eq % '__lt__' -def _not_op(op, other): - # "not a < b" handles "a >= b" - # "not a <= b" handles "a > b" - # "not a >= b" handles "a < b" - # "not a > b" handles "a <= b" - op_result = op(other) +# "not a < b" handles "a >= b" +# "not a <= b" handles "a > b" +# "not a >= b" handles "a < b" +# "not a > b" handles "a <= b" +_not_op = ''' + op_result = self.%s(other) if op_result is NotImplemented: return NotImplemented return not op_result +''' -def _op_or_eq(op, self, other): - # "a < b or a == b" handles "a <= b" - # "a > b or a == b" handles "a >= b" - op_result = op(other) +# "a > b or a == b" handles "a >= b" +# "a < b or a == b" handles "a <= b" +_op_or_eq = ''' + op_result = self.%s(other) if op_result is NotImplemented: return NotImplemented return op_result or self == other +''' -def _not_op_and_not_eq(op, self, other): - # "not (a < b or a == b)" handles "a > b" - # "not a < b and a != b" is equivalent - # "not (a > b or a == b)" handles "a < b" - # "not a > b and a != b" is equivalent - op_result = op(other) +# "not (a < b or a == b)" handles "a > b" +# "not a < b and a != b" is equivalent +# "not (a > b or a == b)" handles "a < b" +# "not a > b and a != b" is equivalent +_not_op_and_not_eq = ''' + op_result = self.%s(other) if op_result is NotImplemented: return NotImplemented return not op_result and self != other +''' -def _not_op_or_eq(op, self, other): - # "not a <= b or a == b" handles "a >= b" - # "not a >= b or a == b" handles "a <= b" - op_result = op(other) +# "not a <= b or a == b" handles "a >= b" +# "not a >= b or a == b" handles "a <= b" +_not_op_or_eq = ''' + op_result = self.%s(other) if op_result is NotImplemented: return NotImplemented return not op_result or self == other +''' -def _op_and_not_eq(op, self, other): - # "a <= b and not a == b" handles "a < b" - # "a >= b and not a == b" handles "a > b" - op_result = op(other) +# "a <= b and not a == b" handles "a < b" +# "a >= b and not a == b" handles "a > b" +_op_and_not_eq = ''' + op_result = self.%s(other) if op_result is NotImplemented: return NotImplemented return op_result and self != other +''' def total_ordering(cls): """Class decorator that fills in missing ordering methods""" convert = { - '__lt__': [('__gt__', lambda self, other: _not_op_and_not_eq(self.__lt__, self, other)), - ('__le__', lambda self, other: _op_or_eq(self.__lt__, self, other)), - ('__ge__', lambda self, other: _not_op(self.__lt__, other))], - '__le__': [('__ge__', lambda self, other: _not_op_or_eq(self.__le__, self, other)), - ('__lt__', lambda self, other: _op_and_not_eq(self.__le__, self, other)), - ('__gt__', lambda self, other: _not_op(self.__le__, other))], - '__gt__': [('__lt__', lambda self, other: _not_op_and_not_eq(self.__gt__, self, other)), - ('__ge__', lambda self, other: _op_or_eq(self.__gt__, self, other)), - ('__le__', lambda self, other: _not_op(self.__gt__, other))], - '__ge__': [('__le__', lambda self, other: _not_op_or_eq(self.__ge__, self, other)), - ('__gt__', lambda self, other: _op_and_not_eq(self.__ge__, self, other)), - ('__lt__', lambda self, other: _not_op(self.__ge__, other))] + '__lt__': {'__gt__': _not_op_and_not_eq, + '__le__': _op_or_eq, + '__ge__': _not_op}, + '__le__': {'__ge__': _not_op_or_eq, + '__lt__': _op_and_not_eq, + '__gt__': _not_op}, + '__gt__': {'__lt__': _not_op_and_not_eq, + '__ge__': _op_or_eq, + '__le__': _not_op}, + '__ge__': {'__le__': _not_op_or_eq, + '__gt__': _op_and_not_eq, + '__lt__': _not_op} } # Find user-defined comparisons (not those inherited from object). roots = [op for op in convert if getattr(cls, op, None) is not getattr(object, op, None)] if not roots: raise ValueError('must define at least one ordering operation: < > <= >=') root = max(roots) # prefer __lt__ to __le__ to __gt__ to __ge__ - for opname, opfunc in convert[root]: + for opname, opfunc in convert[root].items(): if opname not in roots: - opfunc.__name__ = opname + namespace = {} + exec('def %s(self, other):%s' % (opname, opfunc % root), namespace) + opfunc = namespace[opname] + opfunc.__qualname__ = '%s.%s' % (cls.__qualname__, opname) + opfunc.__module__ = cls.__module__ opfunc.__doc__ = getattr(int, opname).__doc__ setattr(cls, opname, opfunc) return cls diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -880,6 +880,24 @@ with self.assertRaises(TypeError): a <= b + def test_pickle(self): + for proto in range(4, pickle.HIGHEST_PROTOCOL + 1): + for name in '__lt__', '__gt__', '__le__', '__ge__': + with self.subTest(method=name, proto=proto): + method = getattr(Orderable_LT, name) + method_copy = pickle.loads(pickle.dumps(method, proto)) + self.assertIs(method_copy, method) + + at functools.total_ordering +class Orderable_LT: + def __init__(self, value): + self.value = value + def __lt__(self, other): + return self.value < other.value + def __eq__(self, other): + return self.value == other.value + + class TestLRU(unittest.TestCase): def test_lru(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -196,6 +196,9 @@ Library ------- +- Issue #23132: Improve performance and introspection support of comparison + methods created by functool.total_ordering. + - Issue #19776: Add a expanduser() method on Path objects. - Issue #23112: Fix SimpleHTTPServer to correctly carry the query string and -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Fri Jan 2 08:57:55 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 02 Jan 2015 08:57:55 +0100 Subject: [Python-checkins] Daily reference leaks (4e85df8b3ea6): sum=3 Message-ID: results for 4e85df8b3ea6 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogYEDrZ1', '-x'] From python-checkins at python.org Fri Jan 2 19:09:57 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 02 Jan 2015 18:09:57 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_conjugate_=27do=27_correctly_?= =?utf-8?b?KGNsb3NlcyAjMjMxNDkp?= Message-ID: <20150102180950.125904.7267@psf.io> https://hg.python.org/peps/rev/0c72bd524aed changeset: 5652:0c72bd524aed user: Benjamin Peterson date: Fri Jan 02 12:09:41 2015 -0600 summary: conjugate 'do' correctly (closes #23149) files: pep-0008.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -394,7 +394,7 @@ ============= In Python, single-quoted strings and double-quoted strings are the -same. This PEP do not make a recommendation for this. Pick a rule +same. This PEP does not make a recommendation for this. Pick a rule and stick to it. When a string contains single or double quote characters, however, use the other one to avoid backslashes in the string. It improves readability. -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 2 19:10:59 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 02 Jan 2015 18:10:59 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_remove_trailing_ws?= Message-ID: <20150102181043.11571.97062@psf.io> https://hg.python.org/peps/rev/aaffbaf67933 changeset: 5653:aaffbaf67933 user: Benjamin Peterson date: Fri Jan 02 12:10:41 2015 -0600 summary: remove trailing ws files: pep-0008.txt | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -967,13 +967,13 @@ - Use ``is not`` operator rather than ``not ... is``. While both expressions are functionally identical, the former is more readable and preferred. - + Yes:: - + if foo is not None: - + No:: - + if not foo is None: - When implementing ordering operations with rich comparisons, it is -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Sat Jan 3 03:11:10 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 03 Jan 2015 02:11:10 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Changes_=25s_to_=25ls_in_w?= =?utf-8?q?printf_in_launcher=2Ec_for_C99_compatibility=2E?= Message-ID: <20150103021105.8747.34424@psf.io> https://hg.python.org/cpython/rev/0a095aa9b5d3 changeset: 94004:0a095aa9b5d3 user: Steve Dower date: Fri Jan 02 18:07:46 2015 -0800 summary: Changes %s to %ls in wprintf in launcher.c for C99 compatibility. files: PC/launcher.c | 102 +++++++++++++++++++------------------- 1 files changed, 51 insertions(+), 51 deletions(-) diff --git a/PC/launcher.c b/PC/launcher.c --- a/PC/launcher.c +++ b/PC/launcher.c @@ -87,13 +87,13 @@ if (rc == 0) { /* a Windows error */ winerror(GetLastError(), win_message, MSGSIZE); if (len >= 0) { - _snwprintf_s(&message[len], MSGSIZE - len, _TRUNCATE, L": %s", + _snwprintf_s(&message[len], MSGSIZE - len, _TRUNCATE, L": %ls", win_message); } } #if !defined(_WINDOWS) - fwprintf(stderr, L"%s\n", message); + fwprintf(stderr, L"%ls\n", message); #else MessageBox(NULL, message, TEXT("Python Launcher is sorry to say ..."), MB_OK); @@ -202,7 +202,7 @@ wchar_t *key_name = (root == HKEY_LOCAL_MACHINE) ? L"HKLM" : L"HKCU"; if (status != ERROR_SUCCESS) - debug(L"locate_pythons_for_key: unable to open PythonCore key in %s\n", + debug(L"locate_pythons_for_key: unable to open PythonCore key in %ls\n", key_name); else { ip = &installed_pythons[num_installed_pythons]; @@ -212,19 +212,19 @@ if (status != ERROR_NO_MORE_ITEMS) { /* unexpected error */ winerror(status, message, MSGSIZE); - debug(L"Can't enumerate registry key for version %s: %s\n", + debug(L"Can't enumerate registry key for version %ls: %ls\n", ip->version, message); } break; } else { _snwprintf_s(ip_path, IP_SIZE, _TRUNCATE, - L"%s\\%s\\InstallPath", CORE_PATH, ip->version); + L"%ls\\%ls\\InstallPath", CORE_PATH, ip->version); status = RegOpenKeyExW(root, ip_path, 0, flags, &ip_key); if (status != ERROR_SUCCESS) { winerror(status, message, MSGSIZE); // Note: 'message' already has a trailing \n - debug(L"%s\\%s: %s", key_name, ip_path, message); + debug(L"%ls\\%ls: %ls", key_name, ip_path, message); continue; } data_size = sizeof(ip->executable) - 1; @@ -233,7 +233,7 @@ RegCloseKey(ip_key); if (status != ERROR_SUCCESS) { winerror(status, message, MSGSIZE); - debug(L"%s\\%s: %s\n", key_name, ip_path, message); + debug(L"%ls\\%ls: %ls\n", key_name, ip_path, message); continue; } if (type == REG_SZ) { @@ -246,27 +246,27 @@ _snwprintf_s(&ip->executable[data_size], MAX_PATH - data_size, MAX_PATH - data_size, - L"%s%s", check, PYTHON_EXECUTABLE); + L"%ls%ls", check, PYTHON_EXECUTABLE); attrs = GetFileAttributesW(ip->executable); if (attrs == INVALID_FILE_ATTRIBUTES) { winerror(GetLastError(), message, MSGSIZE); - debug(L"locate_pythons_for_key: %s: %s", + debug(L"locate_pythons_for_key: %ls: %ls", ip->executable, message); } else if (attrs & FILE_ATTRIBUTE_DIRECTORY) { - debug(L"locate_pythons_for_key: '%s' is a \ + debug(L"locate_pythons_for_key: '%ls' is a \ directory\n", ip->executable, attrs); } else if (find_existing_python(ip->executable)) { - debug(L"locate_pythons_for_key: %s: already \ -found: %s\n", ip->executable); + debug(L"locate_pythons_for_key: %ls: already \ +found: %ls\n", ip->executable); } else { /* check the executable type. */ ok = GetBinaryTypeW(ip->executable, &attrs); if (!ok) { - debug(L"Failure getting binary type: %s\n", + debug(L"Failure getting binary type: %ls\n", ip->executable); } else { @@ -277,7 +277,7 @@ else ip->bits = 0; if (ip->bits == 0) { - debug(L"locate_pythons_for_key: %s: \ + debug(L"locate_pythons_for_key: %ls: \ invalid binary type: %X\n", ip->executable, attrs); } @@ -291,7 +291,7 @@ ip->executable[n + 1] = L'\"'; ip->executable[n + 2] = L'\0'; } - debug(L"locate_pythons_for_key: %s \ + debug(L"locate_pythons_for_key: %ls \ is a %dbit executable\n", ip->executable, ip->bits); ++num_installed_pythons; @@ -397,7 +397,7 @@ DWORD size; /* First, search the environment. */ - _snwprintf_s(configured_value, MSGSIZE, _TRUNCATE, L"py_%s", key); + _snwprintf_s(configured_value, MSGSIZE, _TRUNCATE, L"py_%ls", key); result = get_env(configured_value); if (result == NULL && appdata_ini_path[0]) { /* Not in environment: check local configuration. */ @@ -420,10 +420,10 @@ } } if (result) { - debug(L"found configured value '%s=%s' in %s\n", + debug(L"found configured value '%ls=%ls' in %ls\n", key, result, found_in ? found_in : L"(unknown)"); } else { - debug(L"found no configured value for '%s'\n", key); + debug(L"found no configured value for '%ls'\n", key); } return result; } @@ -449,9 +449,9 @@ } if (*wanted_ver) { result = find_python_by_version(wanted_ver); - debug(L"search for Python version '%s' found ", wanted_ver); + debug(L"search for Python version '%ls' found ", wanted_ver); if (result) { - debug(L"'%s'\n", result->executable); + debug(L"'%ls'\n", result->executable); } else { debug(L"no interpreter\n"); } @@ -467,7 +467,7 @@ result = find_python_by_version(L"3"); debug(L"search for default Python found "); if (result) { - debug(L"version %s at '%s'\n", + debug(L"version %ls at '%ls'\n", result->version, result->executable); } else { debug(L"no interpreter\n"); @@ -505,19 +505,19 @@ plen = GetModuleFileNameW(NULL, wrapped_script_path, MAX_PATH); p = wcsrchr(wrapped_script_path, L'.'); if (p == NULL) { - debug(L"GetModuleFileNameW returned value has no extension: %s\n", + debug(L"GetModuleFileNameW returned value has no extension: %ls\n", wrapped_script_path); - error(RC_NO_SCRIPT, L"Wrapper name '%s' is not valid.", wrapped_script_path); + error(RC_NO_SCRIPT, L"Wrapper name '%ls' is not valid.", wrapped_script_path); } wcsncpy_s(p, MAX_PATH - (p - wrapped_script_path) + 1, SCRIPT_SUFFIX, _TRUNCATE); attrs = GetFileAttributesW(wrapped_script_path); if (attrs == INVALID_FILE_ATTRIBUTES) { - debug(L"File '%s' non-existent\n", wrapped_script_path); - error(RC_NO_SCRIPT, L"Script file '%s' is not present.", wrapped_script_path); + debug(L"File '%ls' non-existent\n", wrapped_script_path); + error(RC_NO_SCRIPT, L"Script file '%ls' is not present.", wrapped_script_path); } - debug(L"Using wrapped script file '%s'\n", wrapped_script_path); + debug(L"Using wrapped script file '%ls'\n", wrapped_script_path); } #endif @@ -579,7 +579,7 @@ GetMessage(&msg, 0, 0, 0); #endif - debug(L"run_child: about to run '%s'\n", cmdline); + debug(L"run_child: about to run '%ls'\n", cmdline); job = CreateJobObject(NULL, NULL); ok = QueryInformationJobObject(job, JobObjectExtendedLimitInformation, &info, sizeof(info), &rc); @@ -611,7 +611,7 @@ ok = CreateProcessW(NULL, cmdline, NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi); if (!ok) - error(RC_CREATE_PROCESS, L"Unable to create process using '%s'", cmdline); + error(RC_CREATE_PROCESS, L"Unable to create process using '%ls'", cmdline); AssignProcessToJobObject(job, pi.hProcess); CloseHandle(pi.hThread); WaitForSingleObjectEx(pi.hProcess, INFINITE, FALSE); @@ -648,11 +648,11 @@ child_command_size); if (no_suffix) _snwprintf_s(child_command, child_command_size, - child_command_size - 1, L"%s %s", + child_command_size - 1, L"%ls %ls", executable, cmdline); else _snwprintf_s(child_command, child_command_size, - child_command_size - 1, L"%s %s %s", + child_command_size - 1, L"%ls %ls %ls", executable, suffix, cmdline); run_child(child_command); free(child_command); @@ -791,7 +791,7 @@ add_command(wchar_t * name, wchar_t * cmdline) { if (num_commands >= MAX_COMMANDS) { - debug(L"can't add %s = '%s': no room\n", name, cmdline); + debug(L"can't add %ls = '%ls': no room\n", name, cmdline); } else { COMMAND * cp = &commands[num_commands++]; @@ -813,14 +813,14 @@ read = GetPrivateProfileStringW(L"commands", NULL, NULL, keynames, MSGSIZE, config_path); if (read == MSGSIZE - 1) { - debug(L"read_commands: %s: not enough space for names\n", config_path); + debug(L"read_commands: %ls: not enough space for names\n", config_path); } key = keynames; while (*key) { read = GetPrivateProfileStringW(L"commands", key, NULL, value, MSGSIZE, config_path); if (read == MSGSIZE - 1) { - debug(L"read_commands: %s: not enough space for %s\n", + debug(L"read_commands: %ls: not enough space for %ls\n", config_path, key); } cmdp = skip_whitespace(value); @@ -1097,7 +1097,7 @@ if ((read >= 4) && (buffer[3] == '\n') && (buffer[2] == '\r')) { ip = find_by_magic((buffer[1] << 8 | buffer[0]) & 0xFFFF); if (ip != NULL) { - debug(L"script file is compiled against Python %s\n", + debug(L"script file is compiled against Python %ls\n", ip->version); invoke_child(ip->executable, NULL, cmdline); } @@ -1200,7 +1200,7 @@ is_virt = parse_shebang(shebang_line, nchars, &command, &suffix, &search); if (command != NULL) { - debug(L"parse_shebang: found command: %s\n", command); + debug(L"parse_shebang: found command: %ls\n", command); if (!is_virt) { invoke_child(command, suffix, cmdline); } @@ -1212,7 +1212,7 @@ } if (wcsncmp(command, L"python", 6)) error(RC_BAD_VIRTUAL_PATH, L"Unknown virtual \ -path '%s'", command); +path '%ls'", command); command += 6; /* skip past "python" */ if (search && ((*command == L'\0') || isspace(*command))) { /* Command is eligible for path search, and there @@ -1220,9 +1220,9 @@ */ debug(L"searching PATH for python executable\n"); cmd = find_on_path(L"python"); - debug(L"Python on path: %s\n", cmd ? cmd->value : L""); + debug(L"Python on path: %ls\n", cmd ? cmd->value : L""); if (cmd) { - debug(L"located python on PATH: %s\n", cmd->value); + debug(L"located python on PATH: %ls\n", cmd->value); invoke_child(cmd->value, suffix, cmdline); /* Exit here, as we have found the command */ return; @@ -1233,14 +1233,14 @@ } if (*command && !validate_version(command)) error(RC_BAD_VIRTUAL_PATH, L"Invalid version \ -specification: '%s'.\nIn the first line of the script, 'python' needs to be \ +specification: '%ls'.\nIn the first line of the script, 'python' needs to be \ followed by a valid version specifier.\nPlease check the documentation.", command); /* TODO could call validate_version(command) */ ip = locate_python(command); if (ip == NULL) { error(RC_NO_PYTHON, L"Requested Python version \ -(%s) is not installed", command); +(%ls) is not installed", command); } else { invoke_child(ip->executable, suffix, cmdline); @@ -1347,17 +1347,17 @@ wcsncpy_s(p, MAX_PATH - plen, L"\\py.ini", _TRUNCATE); attrs = GetFileAttributesW(appdata_ini_path); if (attrs == INVALID_FILE_ATTRIBUTES) { - debug(L"File '%s' non-existent\n", appdata_ini_path); + debug(L"File '%ls' non-existent\n", appdata_ini_path); appdata_ini_path[0] = L'\0'; } else { - debug(L"Using local configuration file '%s'\n", appdata_ini_path); + debug(L"Using local configuration file '%ls'\n", appdata_ini_path); } } plen = GetModuleFileNameW(NULL, launcher_ini_path, MAX_PATH); size = GetFileVersionInfoSizeW(launcher_ini_path, &size); if (size == 0) { winerror(GetLastError(), message, MSGSIZE); - debug(L"GetFileVersionInfoSize failed: %s\n", message); + debug(L"GetFileVersionInfoSize failed: %ls\n", message); } else { version_data = malloc(size); @@ -1381,7 +1381,7 @@ } p = wcsrchr(launcher_ini_path, L'\\'); if (p == NULL) { - debug(L"GetModuleFileNameW returned value has no backslash: %s\n", + debug(L"GetModuleFileNameW returned value has no backslash: %ls\n", launcher_ini_path); launcher_ini_path[0] = L'\0'; } @@ -1390,15 +1390,15 @@ _TRUNCATE); attrs = GetFileAttributesW(launcher_ini_path); if (attrs == INVALID_FILE_ATTRIBUTES) { - debug(L"File '%s' non-existent\n", launcher_ini_path); + debug(L"File '%ls' non-existent\n", launcher_ini_path); launcher_ini_path[0] = L'\0'; } else { - debug(L"Using global configuration file '%s'\n", launcher_ini_path); + debug(L"Using global configuration file '%ls'\n", launcher_ini_path); } } command = skip_me(GetCommandLineW()); - debug(L"Called with command line: %s\n", command); + debug(L"Called with command line: %ls\n", command); #if defined(SCRIPT_WRAPPER) /* The launcher is being used in "script wrapper" mode. @@ -1422,7 +1422,7 @@ wcscpy_s(newcommand, newlen, wrapped_script_path); wcscat_s(newcommand, newlen, L" "); wcscat_s(newcommand, newlen, command); - debug(L"Running wrapped script with command line '%s'\n", newcommand); + debug(L"Running wrapped script with command line '%ls'\n", newcommand); read_commands(); av[0] = wrapped_script_path; av[1] = NULL; @@ -1443,7 +1443,7 @@ if (valid) { ip = locate_python(&p[1]); if (ip == NULL) - error(RC_NO_PYTHON, L"Requested Python version (%s) not \ + error(RC_NO_PYTHON, L"Requested Python version (%ls) not \ installed", &p[1]); command += wcslen(p); command = skip_whitespace(command); @@ -1476,9 +1476,9 @@ get_version_info(version_text, MAX_PATH); fwprintf(stdout, L"\ -Python Launcher for Windows Version %s\n\n", version_text); +Python Launcher for Windows Version %ls\n\n", version_text); fwprintf(stdout, L"\ -usage: %s [ launcher-arguments ] [ python-arguments ] script [ script-arguments ]\n\n", argv[0]); +usage: %ls [ launcher-arguments ] [ python-arguments ] script [ script-arguments ]\n\n", argv[0]); fputws(L"\ Launcher arguments:\n\n\ -2 : Launch the latest Python 2.x version\n\ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 3 03:27:39 2015 From: python-checkins at python.org (chris.angelico) Date: Sat, 03 Jan 2015 02:27:39 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Incorporate_PEP_8_text_from_I?= =?utf-8?q?an_Lee_to_clarify_annotation_policy?= Message-ID: <20150103022738.8743.36822@psf.io> https://hg.python.org/peps/rev/7eb1ddc0291c changeset: 5654:7eb1ddc0291c user: Chris Angelico date: Sat Jan 03 13:22:31 2015 +1100 summary: Incorporate PEP 8 text from Ian Lee to clarify annotation policy files: pep-0008.txt | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-) diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -514,6 +514,19 @@ def complex(real, imag = 0.0): return magic(r = real, i = imag) +- Do use spaces around the ``=`` sign of an annotated function definition. + Additionally, use a single space after the ``:``, as well as a single space + on either side of the ``->`` sign representing an annotated return value. + + Yes: def munge(input: AnyStr): + Yes: def munge(sep: AnyStr = None): + Yes: def munge() -> AnyStr: + Yes: def munge(input: AnyStr, sep: AnyStr = None, limit=1000): + + No: def munge(input: AnyStr=None): + No: def munge(input:AnyStr): + No: def munge(input: AnyStr)->PosInt: + - Compound statements (multiple statements on the same line) are generally discouraged. -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Sat Jan 3 03:27:39 2015 From: python-checkins at python.org (chris.angelico) Date: Sat, 03 Jan 2015 02:27:39 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Formatting_fixes_to_new_PEP8_?= =?utf-8?q?text?= Message-ID: <20150103022738.125880.91415@psf.io> https://hg.python.org/peps/rev/80e4f23d4a96 changeset: 5655:80e4f23d4a96 user: Chris Angelico date: Sat Jan 03 13:27:29 2015 +1100 summary: Formatting fixes to new PEP8 text files: pep-0008.txt | 18 +++++++++++------- 1 files changed, 11 insertions(+), 7 deletions(-) diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -518,14 +518,18 @@ Additionally, use a single space after the ``:``, as well as a single space on either side of the ``->`` sign representing an annotated return value. - Yes: def munge(input: AnyStr): - Yes: def munge(sep: AnyStr = None): - Yes: def munge() -> AnyStr: - Yes: def munge(input: AnyStr, sep: AnyStr = None, limit=1000): + Yes:: - No: def munge(input: AnyStr=None): - No: def munge(input:AnyStr): - No: def munge(input: AnyStr)->PosInt: + def munge(input: AnyStr): + def munge(sep: AnyStr = None): + def munge() -> AnyStr: + def munge(input: AnyStr, sep: AnyStr = None, limit=1000): + + No:: + + def munge(input: AnyStr=None): + def munge(input:AnyStr): + def munge(input: AnyStr)->PosInt: - Compound statements (multiple statements on the same line) are generally discouraged. -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Sat Jan 3 03:31:39 2015 From: python-checkins at python.org (nick.coghlan) Date: Sat, 03 Jan 2015 02:31:39 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_440_updates_based_on_user?= =?utf-8?q?_feedback?= Message-ID: <20150103023139.72567.22977@psf.io> https://hg.python.org/peps/rev/289dbffc16ed changeset: 5656:289dbffc16ed user: Nick Coghlan date: Sat Jan 03 12:28:31 2015 +1000 summary: PEP 440 updates based on user feedback files: pep-0440.txt | 95 ++++++++++++++++++++++++++------------- 1 files changed, 64 insertions(+), 31 deletions(-) diff --git a/pep-0440.txt b/pep-0440.txt --- a/pep-0440.txt +++ b/pep-0440.txt @@ -31,7 +31,7 @@ ======================================== The version representation and comparison scheme described in this PEP is -currently accepted on a provisional basis, as described in PEP 411. +currently accepted on a provisional basis [9]_, as described in PEP 411. This status is based on feedback received on the initial releases of pip 6.0, and setuptools 8.0, which revealed some issues in the specification that @@ -820,8 +820,7 @@ Compatible release ------------------ -A compatible release clause consists of either a version identifier without -any comparison operator or else the compatible release operator ``~=`` +A compatible release clause consists of the compatible release operator ``~=`` and a version identifier. It matches any candidate version that is expected to be compatible with the specified version. @@ -839,11 +838,9 @@ For example, the following groups of version clauses are equivalent:: - 2.2 ~= 2.2 >= 2.2, == 2.* - 1.4.5 ~= 1.4.5 >= 1.4.5, == 1.4.* @@ -851,11 +848,9 @@ compatible release clause as ``V.N.suffix``, then the suffix is ignored when determining the required prefix match:: - 2.2.post3 ~= 2.2.post3 >= 2.2.post3, == 2.* - 1.4.5a4 ~= 1.4.5a4 >= 1.4.5a4, == 1.4.* @@ -863,11 +858,9 @@ degree of forward compatibility in a compatible release clause can be controlled by appending additional zeros to the version specifier:: - 2.2.0 ~= 2.2.0 >= 2.2.0, == 2.2.* - 1.4.5.0 ~= 1.4.5.0 >= 1.4.5.0, == 1.4.5.* @@ -989,20 +982,29 @@ Exclusive ordered comparison ---------------------------- -Exclusive ordered comparisons are similar to inclusive ordered comparisons, -except that the comparison operators are ``<`` and ``>`` and the clause -MUST be effectively interpreted as implying the prefix based version -exclusion clause ``!= V.*``. +The exclusive ordered comparisons ``>`` and ``<`` are similar to the inclusive +ordered comparisons in that they rely on the relative position of the candidate +version and the specified version given the consistent ordering defined by the +standard `Version scheme`_. However, they specifically exclude pre-releases, +post-releases, and local versions of the specified version. -The exclusive ordered comparison ``> V`` MUST NOT match a post-release -or maintenance release of the given version. Maintenance releases can be -permitted by using the clause ``> V.0``, while both post releases and -maintenance releases can be permitted by using the inclusive ordered -comparison ``>= V.post1``. +The exclusive ordered comparison ``>V`` **MUST NOT** allow a post-release +of the given version unless ``V`` itself is a post release. You may mandate +that releases are later than a particular post release, including additional +post releases, by using ``>V.postN``. For example, ``>1.7`` will allow +``1.7.1`` but not ``1.7.0.post1`` and ``>1.7.post2`` will allow ``1.7.1`` +and ``1.7.0.post3`` but not ``1.7.0``. -The exclusive ordered comparison ``< V`` MUST NOT match a pre-release of -the given version, even if acceptance of pre-releases is enabled as -described in the section below. +The exclusive ordered comparison ``>V`` **MUST NOT** match a local version of +the specified version. + +The exclusive ordered comparison ```` -ordered comparison clauses. +added to make it possible to sensibly define compatible release clauses. Support for date based version identifiers @@ -1504,6 +1505,27 @@ gets normalized to a ``_`` to enable easier parsing of the filename. +Summary of changes to \PEP 440 +============================== + +The following changes were made to this PEP based on feedback received after +the initial reference implementation was released in setuptools 8.0 and pip +6.0: + +* The exclusive ordered comparisons were updated to no longer imply a ``!=V.*`` + which was deemed to be surprising behavior which was too hard to accurately + describe. Instead the exclusive ordered comparisons will simply disallow + matching pre-releases, post-releases, and local versions of the specified + version (unless the specified version is itself a pre-release, post-release + or local version). For an extended discussion see the threads on + distutils-sig [6]_ [7]_. + +* The normalized form for release candidates was updated from 'c' to 'rc'. + This change was based on user feedback received when setuptools 8.0 + started applying normalisation to the release metadata generated when + preparing packages for publication on PyPI [8]_. + + References ========== @@ -1525,6 +1547,17 @@ .. [5] Proof of Concept: PEP 440 within pip https://github.com/pypa/pip/pull/1894 +.. [6] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y" + https://mail.python.org/pipermail/distutils-sig/2014-December/025451.html + +.. [7] PEP440: >1.7 vs >=1.7 + https://mail.python.org/pipermail/distutils-sig/2014-December/025507.html + +.. [8] Amend PEP 440 with Wider Feedback on Release Candidates + https://mail.python.org/pipermail/distutils-sig/2014-December/025409.html + +.. [9] Changing the status of PEP 440 to Provisional + https://mail.python.org/pipermail/distutils-sig/2014-December/025412.html Appendix A ========== -- Repository URL: https://hg.python.org/peps From solipsis at pitrou.net Sat Jan 3 09:24:44 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 03 Jan 2015 09:24:44 +0100 Subject: [Python-checkins] Daily reference leaks (0a095aa9b5d3): sum=6 Message-ID: results for 0a095aa9b5d3 on branch "default" -------------------------------------------- test_asyncio leaked [0, 3, 0] memory blocks, sum=3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKI_CsG', '-x'] From python-checkins at python.org Sat Jan 3 09:46:58 2015 From: python-checkins at python.org (ned.deily) Date: Sat, 03 Jan 2015 08:46:58 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Add_missing_UR?= =?utf-8?q?L_link_to_Modernize_docs=2E?= Message-ID: <20150103084653.11569.19487@psf.io> https://hg.python.org/cpython/rev/f4c79656dfd5 changeset: 94006:f4c79656dfd5 branch: 3.4 parent: 94001:50d581f69a73 user: Ned Deily date: Sat Jan 03 00:45:55 2015 -0800 summary: Add missing URL link to Modernize docs. files: Doc/howto/pyporting.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -379,7 +379,7 @@ .. _cheat sheet: http://python-future.org/compatible_idioms.html .. _coverage.py: https://pypi.python.org/pypi/coverage .. _Futurize: http://python-future.org/automatic_conversion.html -.. _Modernize: +.. _Modernize: http://python-modernize.readthedocs.org/en/latest/ .. _Porting to Python 3: http://python3porting.com/ .. _Pylint: https://pypi.python.org/pypi/pylint .. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 3 09:46:58 2015 From: python-checkins at python.org (ned.deily) Date: Sat, 03 Jan 2015 08:46:58 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Add_missing_UR?= =?utf-8?q?L_link_to_Modernize_docs=2E?= Message-ID: <20150103084652.8747.70261@psf.io> https://hg.python.org/cpython/rev/7746f83d8a74 changeset: 94005:7746f83d8a74 branch: 2.7 parent: 94000:4e067cf7e299 user: Ned Deily date: Sat Jan 03 00:45:25 2015 -0800 summary: Add missing URL link to Modernize docs. files: Doc/howto/pyporting.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -379,7 +379,7 @@ .. _cheat sheet: http://python-future.org/compatible_idioms.html .. _coverage.py: https://pypi.python.org/pypi/coverage .. _Futurize: http://python-future.org/automatic_conversion.html -.. _Modernize: +.. _Modernize: http://python-modernize.readthedocs.org/en/latest/ .. _Porting to Python 3: http://python3porting.com/ .. _Pylint: https://pypi.python.org/pypi/pylint .. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 3 09:46:58 2015 From: python-checkins at python.org (ned.deily) Date: Sat, 03 Jan 2015 08:46:58 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Add_missing_URL_link_to_Modernize_docs=2E?= Message-ID: <20150103084653.72567.84035@psf.io> https://hg.python.org/cpython/rev/008b77138424 changeset: 94007:008b77138424 parent: 94004:0a095aa9b5d3 parent: 94006:f4c79656dfd5 user: Ned Deily date: Sat Jan 03 00:46:24 2015 -0800 summary: Add missing URL link to Modernize docs. files: Doc/howto/pyporting.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -379,7 +379,7 @@ .. _cheat sheet: http://python-future.org/compatible_idioms.html .. _coverage.py: https://pypi.python.org/pypi/coverage .. _Futurize: http://python-future.org/automatic_conversion.html -.. _Modernize: +.. _Modernize: http://python-modernize.readthedocs.org/en/latest/ .. _Porting to Python 3: http://python3porting.com/ .. _Pylint: https://pypi.python.org/pypi/pylint .. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 3 11:20:43 2015 From: python-checkins at python.org (donald.stufft) Date: Sat, 03 Jan 2015 10:20:43 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Upgrade_the_bu?= =?utf-8?q?ndled_pip_to_6=2E0=2E6_and_the_bundled_setuptools_to_11=2E0?= Message-ID: <20150103102032.125892.85646@psf.io> https://hg.python.org/cpython/rev/26e1b78448f9 changeset: 94008:26e1b78448f9 branch: 3.4 parent: 94006:f4c79656dfd5 user: Donald Stufft date: Sat Jan 03 05:20:23 2015 -0500 summary: Upgrade the bundled pip to 6.0.6 and the bundled setuptools to 11.0 files: Lib/ensurepip/__init__.py | 4 ++-- Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl | Bin 5 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -8,9 +8,9 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "8.2.1" +_SETUPTOOLS_VERSION = "11.0" -_PIP_VERSION = "6.0.2" +_PIP_VERSION = "6.0.6" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl deleted file mode 100644 index 312467d98be6893a4a965af2e2f53b63188676c0..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..e2be1055f375447e667e34ce2ce018ec41a12221 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..ceeaa3cd5c326495156cb90ed3934ee5b712ad2e GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl deleted file mode 100644 index fa3a6a54053285d701c58cae779ebd9a8a992118..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 3 11:21:36 2015 From: python-checkins at python.org (donald.stufft) Date: Sat, 03 Jan 2015 10:21:36 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Update_bundled_pip_and_setuptools_to_6=2E0=2E6_and_11=2E?= =?utf-8?q?0=2E?= Message-ID: <20150103102129.22419.99748@psf.io> https://hg.python.org/cpython/rev/44cc97e8ec1b changeset: 94009:44cc97e8ec1b parent: 94007:008b77138424 parent: 94008:26e1b78448f9 user: Donald Stufft date: Sat Jan 03 05:21:23 2015 -0500 summary: Update bundled pip and setuptools to 6.0.6 and 11.0. files: Lib/ensurepip/__init__.py | 4 ++-- Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl | Bin 5 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -8,9 +8,9 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "8.2.1" +_SETUPTOOLS_VERSION = "11.0" -_PIP_VERSION = "6.0.2" +_PIP_VERSION = "6.0.6" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl deleted file mode 100644 index 312467d98be6893a4a965af2e2f53b63188676c0..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..e2be1055f375447e667e34ce2ce018ec41a12221 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..ceeaa3cd5c326495156cb90ed3934ee5b712ad2e GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl deleted file mode 100644 index fa3a6a54053285d701c58cae779ebd9a8a992118..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 3 11:23:47 2015 From: python-checkins at python.org (donald.stufft) Date: Sat, 03 Jan 2015 10:23:47 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Update_bundled?= =?utf-8?q?_pip_and_setuptools_to_6=2E0=2E6_and_11=2E0?= Message-ID: <20150103102345.72557.71261@psf.io> https://hg.python.org/cpython/rev/1346a6013127 changeset: 94010:1346a6013127 branch: 2.7 parent: 94005:7746f83d8a74 user: Donald Stufft date: Sat Jan 03 05:23:39 2015 -0500 summary: Update bundled pip and setuptools to 6.0.6 and 11.0 files: Lib/ensurepip/__init__.py | 4 ++-- Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl | Bin 5 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -12,9 +12,9 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "8.2.1" +_SETUPTOOLS_VERSION = "11.0" -_PIP_VERSION = "6.0.2" +_PIP_VERSION = "6.0.6" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.2-py2.py3-none-any.whl deleted file mode 100644 index 312467d98be6893a4a965af2e2f53b63188676c0..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..e2be1055f375447e667e34ce2ce018ec41a12221 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..ceeaa3cd5c326495156cb90ed3934ee5b712ad2e GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-8.2.1-py2.py3-none-any.whl deleted file mode 100644 index fa3a6a54053285d701c58cae779ebd9a8a992118..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 3 23:17:30 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 03 Jan 2015 22:17:30 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323143=3A_Remove_c?= =?utf-8?q?ompatibility_with_OpenSSLs_older_than_0=2E9=2E8=2E?= Message-ID: <20150103221729.72575.62261@psf.io> https://hg.python.org/cpython/rev/e9f05a4a5f16 changeset: 94011:e9f05a4a5f16 parent: 94009:44cc97e8ec1b user: Antoine Pitrou date: Sat Jan 03 23:17:23 2015 +0100 summary: Issue #23143: Remove compatibility with OpenSSLs older than 0.9.8. (the last 0.9.7 release was in 2007) files: Misc/NEWS | 2 + Modules/_ssl.c | 53 -------------------------------------- 2 files changed, 2 insertions(+), 53 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -196,6 +196,8 @@ Library ------- +- Issue #23143: Remove compatibility with OpenSSLs older than 0.9.8. + - Issue #23132: Improve performance and introspection support of comparison methods created by functool.total_ordering. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -162,13 +162,6 @@ #define X509_NAME_MAXLEN 256 -/* RAND_* APIs got added to OpenSSL in 0.9.5 */ -#if OPENSSL_VERSION_NUMBER >= 0x0090500fL -# define HAVE_OPENSSL_RAND 1 -#else -# undef HAVE_OPENSSL_RAND -#endif - /* SSL_CTX_clear_options() and SSL_clear_options() were first added in * OpenSSL 0.9.8m but do not appear in some 0.9.9-dev versions such the * 0.9.9 from "May 2008" that NetBSD 5.0 uses. */ @@ -182,28 +175,6 @@ * older SSL, but let's be safe */ #define PySSL_CB_MAXLEN 128 -/* SSL_get_finished got added to OpenSSL in 0.9.5 */ -#if OPENSSL_VERSION_NUMBER >= 0x0090500fL -# define HAVE_OPENSSL_FINISHED 1 -#else -# define HAVE_OPENSSL_FINISHED 0 -#endif - -/* ECDH support got added to OpenSSL in 0.9.8 */ -#if OPENSSL_VERSION_NUMBER < 0x0090800fL && !defined(OPENSSL_NO_ECDH) -# define OPENSSL_NO_ECDH -#endif - -/* compression support got added to OpenSSL in 0.9.8 */ -#if OPENSSL_VERSION_NUMBER < 0x0090800fL && !defined(OPENSSL_NO_COMP) -# define OPENSSL_NO_COMP -#endif - -/* X509_VERIFY_PARAM got added to OpenSSL in 0.9.8 */ -#if OPENSSL_VERSION_NUMBER >= 0x0090800fL -# define HAVE_OPENSSL_VERIFY_PARAM -#endif - typedef struct { PyObject_HEAD @@ -817,12 +788,7 @@ char buf[2048]; char *vptr; int len; - /* Issue #2973: ASN1_item_d2i() API changed in OpenSSL 0.9.6m */ -#if OPENSSL_VERSION_NUMBER >= 0x009060dfL const unsigned char *p; -#else - unsigned char *p; -#endif if (certificate == NULL) return peer_alt_names; @@ -1998,7 +1964,6 @@ Does the SSL shutdown handshake with the remote end, and returns\n\ the underlying socket object."); -#if HAVE_OPENSSL_FINISHED static PyObject * PySSL_tls_unique_cb(PySSLSocket *self) { @@ -2031,8 +1996,6 @@ \n\ If the TLS handshake is not yet complete, None is returned"); -#endif /* HAVE_OPENSSL_FINISHED */ - static PyGetSetDef ssl_getsetlist[] = { {"context", (getter) PySSL_get_context, (setter) PySSL_set_context, PySSL_set_context_doc}, @@ -2063,10 +2026,8 @@ {"compression", (PyCFunction)PySSL_compression, METH_NOARGS}, {"shutdown", (PyCFunction)PySSL_SSLshutdown, METH_NOARGS, PySSL_SSLshutdown_doc}, -#if HAVE_OPENSSL_FINISHED {"tls_unique_cb", (PyCFunction)PySSL_tls_unique_cb, METH_NOARGS, PySSL_tls_unique_cb_doc}, -#endif {NULL, NULL} }; @@ -2380,7 +2341,6 @@ return 0; } -#ifdef HAVE_OPENSSL_VERIFY_PARAM static PyObject * get_verify_flags(PySSLContext *self, void *c) { @@ -2418,7 +2378,6 @@ } return 0; } -#endif static PyObject * get_options(PySSLContext *self, void *c) @@ -3303,10 +3262,8 @@ (setter) set_check_hostname, NULL}, {"options", (getter) get_options, (setter) set_options, NULL}, -#ifdef HAVE_OPENSSL_VERIFY_PARAM {"verify_flags", (getter) get_verify_flags, (setter) set_verify_flags, NULL}, -#endif {"verify_mode", (getter) get_verify_mode, (setter) set_verify_mode, NULL}, {NULL}, /* sentinel */ @@ -3606,8 +3563,6 @@ }; -#ifdef HAVE_OPENSSL_RAND - /* helper routines for seeding the SSL PRNG */ static PyObject * PySSL_RAND_add(PyObject *self, PyObject *args) @@ -3745,8 +3700,6 @@ fails or if it does not provide enough data to seed PRNG."); #endif /* HAVE_RAND_EGD */ -#endif /* HAVE_OPENSSL_RAND */ - PyDoc_STRVAR(PySSL_get_default_verify_paths_doc, "get_default_verify_paths() -> tuple\n\ @@ -4132,7 +4085,6 @@ static PyMethodDef PySSL_methods[] = { {"_test_decode_cert", PySSL_test_decode_certificate, METH_VARARGS}, -#ifdef HAVE_OPENSSL_RAND {"RAND_add", PySSL_RAND_add, METH_VARARGS, PySSL_RAND_add_doc}, {"RAND_bytes", PySSL_RAND_bytes, METH_VARARGS, @@ -4145,7 +4097,6 @@ #endif {"RAND_status", (PyCFunction)PySSL_RAND_status, METH_NOARGS, PySSL_RAND_status_doc}, -#endif {"get_default_verify_paths", (PyCFunction)PySSL_get_default_verify_paths, METH_NOARGS, PySSL_get_default_verify_paths_doc}, #ifdef _MSC_VER @@ -4500,11 +4451,7 @@ Py_INCREF(r); PyModule_AddObject(m, "HAS_SNI", r); -#if HAVE_OPENSSL_FINISHED r = Py_True; -#else - r = Py_False; -#endif Py_INCREF(r); PyModule_AddObject(m, "HAS_TLS_UNIQUE", r); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 3 23:21:27 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 03 Jan 2015 22:21:27 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323143=3A_Remove_c?= =?utf-8?q?ompatibility_with_OpenSSLs_older_than_0=2E9=2E8=2E?= Message-ID: <20150103222126.8749.38486@psf.io> https://hg.python.org/cpython/rev/37c6fd09f71f changeset: 94012:37c6fd09f71f user: Antoine Pitrou date: Sat Jan 03 23:21:21 2015 +0100 summary: Issue #23143: Remove compatibility with OpenSSLs older than 0.9.8. (now the hashlib module) files: Modules/_hashopenssl.c | 12 ------------ 1 files changed, 0 insertions(+), 12 deletions(-) diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -31,10 +31,6 @@ #define HASH_OBJ_CONSTRUCTOR 0 #endif -/* Minimum OpenSSL version needed to support sha224 and higher. */ -#if defined(OPENSSL_VERSION_NUMBER) && (OPENSSL_VERSION_NUMBER >= 0x00908000) -#define _OPENSSL_SUPPORTS_SHA2 -#endif typedef struct { PyObject_HEAD @@ -56,12 +52,10 @@ DEFINE_CONSTS_FOR_NEW(md5) DEFINE_CONSTS_FOR_NEW(sha1) -#ifdef _OPENSSL_SUPPORTS_SHA2 DEFINE_CONSTS_FOR_NEW(sha224) DEFINE_CONSTS_FOR_NEW(sha256) DEFINE_CONSTS_FOR_NEW(sha384) DEFINE_CONSTS_FOR_NEW(sha512) -#endif static EVPobject * @@ -798,12 +792,10 @@ GEN_CONSTRUCTOR(md5) GEN_CONSTRUCTOR(sha1) -#ifdef _OPENSSL_SUPPORTS_SHA2 GEN_CONSTRUCTOR(sha224) GEN_CONSTRUCTOR(sha256) GEN_CONSTRUCTOR(sha384) GEN_CONSTRUCTOR(sha512) -#endif /* List of functions exported by this module */ @@ -815,12 +807,10 @@ #endif CONSTRUCTOR_METH_DEF(md5), CONSTRUCTOR_METH_DEF(sha1), -#ifdef _OPENSSL_SUPPORTS_SHA2 CONSTRUCTOR_METH_DEF(sha224), CONSTRUCTOR_METH_DEF(sha256), CONSTRUCTOR_METH_DEF(sha384), CONSTRUCTOR_METH_DEF(sha512), -#endif {NULL, NULL} /* Sentinel */ }; @@ -877,11 +867,9 @@ /* these constants are used by the convenience constructors */ INIT_CONSTRUCTOR_CONSTANTS(md5); INIT_CONSTRUCTOR_CONSTANTS(sha1); -#ifdef _OPENSSL_SUPPORTS_SHA2 INIT_CONSTRUCTOR_CONSTANTS(sha224); INIT_CONSTRUCTOR_CONSTANTS(sha256); INIT_CONSTRUCTOR_CONSTANTS(sha384); INIT_CONSTRUCTOR_CONSTANTS(sha512); -#endif return m; } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 05:33:26 2015 From: python-checkins at python.org (zach.ware) Date: Sun, 04 Jan 2015 04:33:26 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2323154=3A_Fix_unn?= =?utf-8?q?ecessary_recompilation_of_OpenSSL_on_Windows?= Message-ID: <20150104043326.22421.41033@psf.io> https://hg.python.org/cpython/rev/d53506fe31e1 changeset: 94013:d53506fe31e1 user: Zachary Ware date: Sat Jan 03 22:33:10 2015 -0600 summary: Closes #23154: Fix unnecessary recompilation of OpenSSL on Windows files: PCbuild/libeay.vcxproj | 6 ------ PCbuild/openssl.props | 6 ++++++ PCbuild/ssleay.vcxproj | 6 ------ 3 files changed, 6 insertions(+), 12 deletions(-) diff --git a/PCbuild/libeay.vcxproj b/PCbuild/libeay.vcxproj --- a/PCbuild/libeay.vcxproj +++ b/PCbuild/libeay.vcxproj @@ -42,12 +42,6 @@ - - - StaticLibrary - $(opensslDir)tmp32\libeay\ - $(opensslDir)tmp64\libeay\ - diff --git a/PCbuild/openssl.props b/PCbuild/openssl.props --- a/PCbuild/openssl.props +++ b/PCbuild/openssl.props @@ -2,6 +2,12 @@ + + StaticLibrary + $(opensslDir)tmp\$(ArchName)_$(Configuration)\$(ProjectName)\ + $(opensslDir)tmp\$(ArchName)\$(ProjectName)\ + + diff --git a/PCbuild/ssleay.vcxproj b/PCbuild/ssleay.vcxproj --- a/PCbuild/ssleay.vcxproj +++ b/PCbuild/ssleay.vcxproj @@ -42,12 +42,6 @@ - - - StaticLibrary - $(opensslDir)tmp32\ssleay\ - $(opensslDir)tmp64\ssleay\ - -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Sun Jan 4 09:22:10 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 04 Jan 2015 09:22:10 +0100 Subject: [Python-checkins] Daily reference leaks (37c6fd09f71f): sum=3 Message-ID: results for 37c6fd09f71f on branch "default" -------------------------------------------- test_collections leaked [-2, 2, 0] references, sum=0 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogYTpcfK', '-x'] From python-checkins at python.org Sun Jan 4 09:37:21 2015 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 04 Jan 2015 08:37:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_fix_issue23157_-_time=5Fhashlib_hadn=27t_been_ported_to_?= =?utf-8?q?Python_3=2E?= Message-ID: <20150104083711.72565.42292@psf.io> https://hg.python.org/cpython/rev/b96985753613 changeset: 94015:b96985753613 parent: 94013:d53506fe31e1 parent: 94014:badb7e319ed0 user: Gregory P. Smith date: Sun Jan 04 00:36:59 2015 -0800 summary: fix issue23157 - time_hashlib hadn't been ported to Python 3. files: Lib/test/time_hashlib.py | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/test/time_hashlib.py b/Lib/test/time_hashlib.py --- a/Lib/test/time_hashlib.py +++ b/Lib/test/time_hashlib.py @@ -1,7 +1,8 @@ # It's intended that this script be run by hand. It runs speed tests on # hashlib functions; it does not test for correctness. -import sys, time +import sys +import time import hashlib @@ -9,8 +10,8 @@ raise RuntimeError("eek, creatorFunc not overridden") def test_scaled_msg(scale, name): - iterations = 106201/scale * 20 - longStr = 'Z'*scale + iterations = 106201//scale * 20 + longStr = b'Z'*scale localCF = creatorFunc start = time.time() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 09:37:21 2015 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 04 Jan 2015 08:37:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_fix_issue23157?= =?utf-8?q?_-_time=5Fhashlib_hadn=27t_been_ported_to_Python_3=2E?= Message-ID: <20150104083711.11579.88013@psf.io> https://hg.python.org/cpython/rev/badb7e319ed0 changeset: 94014:badb7e319ed0 branch: 3.4 parent: 94008:26e1b78448f9 user: Gregory P. Smith date: Sun Jan 04 00:36:04 2015 -0800 summary: fix issue23157 - time_hashlib hadn't been ported to Python 3. files: Lib/test/time_hashlib.py | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/test/time_hashlib.py b/Lib/test/time_hashlib.py --- a/Lib/test/time_hashlib.py +++ b/Lib/test/time_hashlib.py @@ -1,7 +1,8 @@ # It's intended that this script be run by hand. It runs speed tests on # hashlib functions; it does not test for correctness. -import sys, time +import sys +import time import hashlib @@ -9,8 +10,8 @@ raise RuntimeError("eek, creatorFunc not overridden") def test_scaled_msg(scale, name): - iterations = 106201/scale * 20 - longStr = 'Z'*scale + iterations = 106201//scale * 20 + longStr = b'Z'*scale localCF = creatorFunc start = time.time() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 17:20:44 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 04 Jan 2015 16:20:44 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_make_SSLv23_th?= =?utf-8?q?e_default_version_in_ftplib_=28closes_=2323111=29?= Message-ID: <20150104162021.125896.97738@psf.io> https://hg.python.org/cpython/rev/98ee845a139a changeset: 94016:98ee845a139a branch: 2.7 parent: 94010:1346a6013127 user: Benjamin Peterson date: Sun Jan 04 10:20:16 2015 -0600 summary: make SSLv23 the default version in ftplib (closes #23111) files: Doc/library/ftplib.rst | 2 +- Lib/ftplib.py | 4 ++-- Misc/NEWS | 2 ++ 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -384,7 +384,7 @@ .. attribute:: FTP_TLS.ssl_version - The SSL version to use (defaults to *TLSv1*). + The SSL version to use (defaults to :attr:`ssl.PROTOCOL_SSLv23`). .. method:: FTP_TLS.auth() diff --git a/Lib/ftplib.py b/Lib/ftplib.py --- a/Lib/ftplib.py +++ b/Lib/ftplib.py @@ -638,7 +638,7 @@ '221 Goodbye.' >>> ''' - ssl_version = ssl.PROTOCOL_TLSv1 + ssl_version = ssl.PROTOCOL_SSLv23 def __init__(self, host='', user='', passwd='', acct='', keyfile=None, certfile=None, timeout=_GLOBAL_DEFAULT_TIMEOUT): @@ -656,7 +656,7 @@ '''Set up secure control connection by using TLS/SSL.''' if isinstance(self.sock, ssl.SSLSocket): raise ValueError("Already using TLS") - if self.ssl_version == ssl.PROTOCOL_TLSv1: + if self.ssl_version >= ssl.PROTOCOL_SSLv23: resp = self.voidcmd('AUTH TLS') else: resp = self.voidcmd('AUTH SSL') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Library ------- +- Issue #23111: Maximize compatibility in protocol versions of ftplib.FTP_TLS. + - Issue #23112: Fix SimpleHTTPServer to correctly carry the query string and fragment when it redirects to add a trailing slash. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 22:36:56 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 04 Jan 2015 21:36:56 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_allow_a_SSLCon?= =?utf-8?q?text_to_be_given_to_ftplib=2EFTP=5FTLS?= Message-ID: <20150104213653.8765.37503@psf.io> https://hg.python.org/cpython/rev/e8342b3154d1 changeset: 94017:e8342b3154d1 branch: 2.7 user: Benjamin Peterson date: Sun Jan 04 15:36:31 2015 -0600 summary: allow a SSLContext to be given to ftplib.FTP_TLS files: Doc/library/ftplib.rst | 18 +++- Lib/ftplib.py | 22 ++++- Lib/test/test_ftplib.py | 115 +++++++++++++++++++++------ Misc/NEWS | 2 + 4 files changed, 120 insertions(+), 37 deletions(-) diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -55,18 +55,26 @@ *timeout* was added. -.. class:: FTP_TLS([host[, user[, passwd[, acct[, keyfile[, certfile[, timeout]]]]]]]) +.. class:: FTP_TLS([host[, user[, passwd[, acct[, keyfile[, certfile[, context[, timeout]]]]]]]]) - A :class:`FTP` subclass which adds TLS support to FTP as described in + A :class:`FTP` subclass which adds TLS support to FTP as described in :rfc:`4217`. Connect as usual to port 21 implicitly securing the FTP control connection before authenticating. Securing the data connection requires the user to - explicitly ask for it by calling the :meth:`prot_p` method. - *keyfile* and *certfile* are optional -- they can contain a PEM formatted - private key and certificate chain file name for the SSL connection. + explicitly ask for it by calling the :meth:`prot_p` method. *context* + is a :class:`ssl.SSLContext` object which allows bundling SSL configuration + options, certificates and private keys into a single (potentially + long-lived) structure. Please read :ref:`ssl-security` for best practices. + + *keyfile* and *certfile* are a legacy alternative to *context* -- they + can point to PEM-formatted private key and certificate chain files + (respectively) for the SSL connection. .. versionadded:: 2.7 + .. versionchanged:: 2.7.10 + The *context* parameter was added. + Here's a sample session using the :class:`FTP_TLS` class: >>> from ftplib import FTP_TLS diff --git a/Lib/ftplib.py b/Lib/ftplib.py --- a/Lib/ftplib.py +++ b/Lib/ftplib.py @@ -641,9 +641,21 @@ ssl_version = ssl.PROTOCOL_SSLv23 def __init__(self, host='', user='', passwd='', acct='', keyfile=None, - certfile=None, timeout=_GLOBAL_DEFAULT_TIMEOUT): + certfile=None, context=None, + timeout=_GLOBAL_DEFAULT_TIMEOUT, source_address=None): + if context is not None and keyfile is not None: + raise ValueError("context and keyfile arguments are mutually " + "exclusive") + if context is not None and certfile is not None: + raise ValueError("context and certfile arguments are mutually " + "exclusive") self.keyfile = keyfile self.certfile = certfile + if context is None: + context = ssl._create_stdlib_context(self.ssl_version, + certfile=certfile, + keyfile=keyfile) + self.context = context self._prot_p = False FTP.__init__(self, host, user, passwd, acct, timeout) @@ -660,8 +672,8 @@ resp = self.voidcmd('AUTH TLS') else: resp = self.voidcmd('AUTH SSL') - self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile, - ssl_version=self.ssl_version) + self.sock = self.context.wrap_socket(self.sock, + server_hostname=self.host) self.file = self.sock.makefile(mode='rb') return resp @@ -692,8 +704,8 @@ def ntransfercmd(self, cmd, rest=None): conn, size = FTP.ntransfercmd(self, cmd, rest) if self._prot_p: - conn = ssl.wrap_socket(conn, self.keyfile, self.certfile, - ssl_version=self.ssl_version) + conn = self.context.wrap_socket(conn, + server_hostname=self.host) return conn, size def retrbinary(self, cmd, callback, blocksize=8192, rest=None): diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -20,7 +20,7 @@ from test.test_support import HOST, HOSTv6 threading = test_support.import_module('threading') - +TIMEOUT = 3 # the dummy data returned by server over the data channel when # RETR, LIST and NLST commands are issued RETR_DATA = 'abcde12345\r\n' * 1000 @@ -223,6 +223,7 @@ self.active = False self.active_lock = threading.Lock() self.host, self.port = self.socket.getsockname()[:2] + self.handler_instance = None def start(self): assert not self.active @@ -246,8 +247,7 @@ def handle_accept(self): conn, addr = self.accept() - self.handler = self.handler(conn) - self.close() + self.handler_instance = self.handler(conn) def handle_connect(self): self.close() @@ -262,7 +262,8 @@ if ssl is not None: - CERTFILE = os.path.join(os.path.dirname(__file__), "keycert.pem") + CERTFILE = os.path.join(os.path.dirname(__file__), "keycert3.pem") + CAFILE = os.path.join(os.path.dirname(__file__), "pycacert.pem") class SSLConnection(object, asyncore.dispatcher): """An asyncore.dispatcher subclass supporting TLS/SSL.""" @@ -271,23 +272,25 @@ _ssl_closing = False def secure_connection(self): - self.socket = ssl.wrap_socket(self.socket, suppress_ragged_eofs=False, - certfile=CERTFILE, server_side=True, - do_handshake_on_connect=False, - ssl_version=ssl.PROTOCOL_SSLv23) + socket = ssl.wrap_socket(self.socket, suppress_ragged_eofs=False, + certfile=CERTFILE, server_side=True, + do_handshake_on_connect=False, + ssl_version=ssl.PROTOCOL_SSLv23) + self.del_channel() + self.set_socket(socket) self._ssl_accepting = True def _do_ssl_handshake(self): try: self.socket.do_handshake() - except ssl.SSLError, err: + except ssl.SSLError as err: if err.args[0] in (ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE): return elif err.args[0] == ssl.SSL_ERROR_EOF: return self.handle_close() raise - except socket.error, err: + except socket.error as err: if err.args[0] == errno.ECONNABORTED: return self.handle_close() else: @@ -297,18 +300,21 @@ self._ssl_closing = True try: self.socket = self.socket.unwrap() - except ssl.SSLError, err: + except ssl.SSLError as err: if err.args[0] in (ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE): return - except socket.error, err: + except socket.error as err: # Any "socket error" corresponds to a SSL_ERROR_SYSCALL return # from OpenSSL's SSL_shutdown(), corresponding to a # closed socket condition. See also: # http://www.mail-archive.com/openssl-users at openssl.org/msg60710.html pass self._ssl_closing = False - super(SSLConnection, self).close() + if getattr(self, '_ccc', False) is False: + super(SSLConnection, self).close() + else: + pass def handle_read_event(self): if self._ssl_accepting: @@ -329,7 +335,7 @@ def send(self, data): try: return super(SSLConnection, self).send(data) - except ssl.SSLError, err: + except ssl.SSLError as err: if err.args[0] in (ssl.SSL_ERROR_EOF, ssl.SSL_ERROR_ZERO_RETURN, ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE): @@ -339,13 +345,13 @@ def recv(self, buffer_size): try: return super(SSLConnection, self).recv(buffer_size) - except ssl.SSLError, err: + except ssl.SSLError as err: if err.args[0] in (ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE): - return '' + return b'' if err.args[0] in (ssl.SSL_ERROR_EOF, ssl.SSL_ERROR_ZERO_RETURN): self.handle_close() - return '' + return b'' raise def handle_error(self): @@ -355,6 +361,8 @@ if (isinstance(self.socket, ssl.SSLSocket) and self.socket._sslobj is not None): self._do_ssl_shutdown() + else: + super(SSLConnection, self).close() class DummyTLS_DTPHandler(SSLConnection, DummyDTPHandler): @@ -462,12 +470,12 @@ def test_rename(self): self.client.rename('a', 'b') - self.server.handler.next_response = '200' + self.server.handler_instance.next_response = '200' self.assertRaises(ftplib.error_reply, self.client.rename, 'a', 'b') def test_delete(self): self.client.delete('foo') - self.server.handler.next_response = '199' + self.server.handler_instance.next_response = '199' self.assertRaises(ftplib.error_reply, self.client.delete, 'foo') def test_size(self): @@ -515,7 +523,7 @@ def test_storbinary(self): f = StringIO.StringIO(RETR_DATA) self.client.storbinary('stor', f) - self.assertEqual(self.server.handler.last_received_data, RETR_DATA) + self.assertEqual(self.server.handler_instance.last_received_data, RETR_DATA) # test new callback arg flag = [] f.seek(0) @@ -527,12 +535,12 @@ for r in (30, '30'): f.seek(0) self.client.storbinary('stor', f, rest=r) - self.assertEqual(self.server.handler.rest, str(r)) + self.assertEqual(self.server.handler_instance.rest, str(r)) def test_storlines(self): f = StringIO.StringIO(RETR_DATA.replace('\r\n', '\n')) self.client.storlines('stor', f) - self.assertEqual(self.server.handler.last_received_data, RETR_DATA) + self.assertEqual(self.server.handler_instance.last_received_data, RETR_DATA) # test new callback arg flag = [] f.seek(0) @@ -551,14 +559,14 @@ def test_makeport(self): self.client.makeport() # IPv4 is in use, just make sure send_eprt has not been used - self.assertEqual(self.server.handler.last_received_cmd, 'port') + self.assertEqual(self.server.handler_instance.last_received_cmd, 'port') def test_makepasv(self): host, port = self.client.makepasv() conn = socket.create_connection((host, port), 10) conn.close() # IPv4 is in use, just make sure send_epsv has not been used - self.assertEqual(self.server.handler.last_received_cmd, 'pasv') + self.assertEqual(self.server.handler_instance.last_received_cmd, 'pasv') def test_line_too_long(self): self.assertRaises(ftplib.Error, self.client.sendcmd, @@ -600,13 +608,13 @@ def test_makeport(self): self.client.makeport() - self.assertEqual(self.server.handler.last_received_cmd, 'eprt') + self.assertEqual(self.server.handler_instance.last_received_cmd, 'eprt') def test_makepasv(self): host, port = self.client.makepasv() conn = socket.create_connection((host, port), 10) conn.close() - self.assertEqual(self.server.handler.last_received_cmd, 'epsv') + self.assertEqual(self.server.handler_instance.last_received_cmd, 'epsv') def test_transfer(self): def retr(): @@ -642,7 +650,7 @@ def setUp(self): self.server = DummyTLS_FTPServer((HOST, 0)) self.server.start() - self.client = ftplib.FTP_TLS(timeout=10) + self.client = ftplib.FTP_TLS(timeout=TIMEOUT) self.client.connect(self.server.host, self.server.port) def tearDown(self): @@ -695,6 +703,59 @@ finally: self.client.ssl_version = ssl.PROTOCOL_TLSv1 + def test_context(self): + self.client.quit() + ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + self.assertRaises(ValueError, ftplib.FTP_TLS, keyfile=CERTFILE, + context=ctx) + self.assertRaises(ValueError, ftplib.FTP_TLS, certfile=CERTFILE, + context=ctx) + self.assertRaises(ValueError, ftplib.FTP_TLS, certfile=CERTFILE, + keyfile=CERTFILE, context=ctx) + + self.client = ftplib.FTP_TLS(context=ctx, timeout=TIMEOUT) + self.client.connect(self.server.host, self.server.port) + self.assertNotIsInstance(self.client.sock, ssl.SSLSocket) + self.client.auth() + self.assertIs(self.client.sock.context, ctx) + self.assertIsInstance(self.client.sock, ssl.SSLSocket) + + self.client.prot_p() + sock = self.client.transfercmd('list') + try: + self.assertIs(sock.context, ctx) + self.assertIsInstance(sock, ssl.SSLSocket) + finally: + sock.close() + + def test_check_hostname(self): + self.client.quit() + ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + ctx.verify_mode = ssl.CERT_REQUIRED + ctx.check_hostname = True + ctx.load_verify_locations(CAFILE) + self.client = ftplib.FTP_TLS(context=ctx, timeout=TIMEOUT) + + # 127.0.0.1 doesn't match SAN + self.client.connect(self.server.host, self.server.port) + with self.assertRaises(ssl.CertificateError): + self.client.auth() + # exception quits connection + + self.client.connect(self.server.host, self.server.port) + self.client.prot_p() + with self.assertRaises(ssl.CertificateError): + self.client.transfercmd("list").close() + self.client.quit() + + self.client.connect("localhost", self.server.port) + self.client.auth() + self.client.quit() + + self.client.connect("localhost", self.server.port) + self.client.prot_p() + self.client.transfercmd("list").close() + class TestTimeouts(TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Library ------- +- Backport the context argument to ftplib.FTP_TLS. + - Issue #23111: Maximize compatibility in protocol versions of ftplib.FTP_TLS. - Issue #23112: Fix SimpleHTTPServer to correctly carry the query string and -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 23:07:07 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 04 Jan 2015 22:07:07 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_add_some_overf?= =?utf-8?q?low_checks_before_multiplying_=28closes_=2323165=29?= Message-ID: <20150104220625.72551.67814@psf.io> https://hg.python.org/cpython/rev/1ce98e85929d changeset: 94018:1ce98e85929d branch: 3.2 parent: 93995:1806a00fd6b8 user: Benjamin Peterson date: Sun Jan 04 16:03:17 2015 -0600 summary: add some overflow checks before multiplying (closes #23165) files: Misc/NEWS | 3 +++ Python/fileutils.c | 16 +++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #23165: Perform overflow checks before allocating memory in the + _Py_char2wchar function. + - Issue #19529: Fix a potential crash in converting Unicode objects to wchar_t when Py_UNICODE is 4 bytes but wchar_t is 2 bytes, for example on AIX. diff --git a/Python/fileutils.c b/Python/fileutils.c --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -169,8 +169,11 @@ wchar_t *res; unsigned char *in; wchar_t *out; + size_t argsize = strlen(arg) + 1; - res = PyMem_Malloc((strlen(arg)+1)*sizeof(wchar_t)); + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + return NULL; + res = PyMem_Malloc(argsize*sizeof(wchar_t)); if (!res) return NULL; @@ -250,10 +253,15 @@ argsize = mbstowcs(NULL, arg, 0); #endif if (argsize != (size_t)-1) { - res = (wchar_t *)PyMem_Malloc((argsize+1)*sizeof(wchar_t)); + if (argsize == PY_SSIZE_T_MAX) + goto oom; + argsize += 1; + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + goto oom; + res = (wchar_t *)PyMem_Malloc(argsize*sizeof(wchar_t)); if (!res) goto oom; - count = mbstowcs(res, arg, argsize+1); + count = mbstowcs(res, arg, argsize); if (count != (size_t)-1) { wchar_t *tmp; /* Only use the result if it contains no @@ -276,6 +284,8 @@ /* Overallocate; as multi-byte characters are in the argument, the actual output could use less memory. */ argsize = strlen(arg) + 1; + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + goto oom; res = (wchar_t*)PyMem_Malloc(argsize*sizeof(wchar_t)); if (!res) goto oom; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 23:07:07 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 04 Jan 2015 22:07:07 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMxNjUp?= Message-ID: <20150104220625.11585.65183@psf.io> https://hg.python.org/cpython/rev/8c4fb312e15d changeset: 94021:8c4fb312e15d parent: 94015:b96985753613 parent: 94020:d45e16b1ed86 user: Benjamin Peterson date: Sun Jan 04 16:06:14 2015 -0600 summary: merge 3.4 (#23165) files: Misc/NEWS | 3 +++ Python/fileutils.c | 16 +++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -193,6 +193,9 @@ exception. In versions prior to 3.5, '#' with 'c' had no effect. Now specifying it is an error. Patch by Torsten Landschoff. +- Issue #23165: Perform overflow checks before allocating memory in the + _Py_char2wchar function. + Library ------- diff --git a/Python/fileutils.c b/Python/fileutils.c --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -220,8 +220,11 @@ wchar_t *res; unsigned char *in; wchar_t *out; + size_t argsize = strlen(arg) + 1; - res = PyMem_RawMalloc((strlen(arg)+1)*sizeof(wchar_t)); + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + return NULL; + res = PyMem_RawMalloc(argsize*sizeof(wchar_t)); if (!res) return NULL; @@ -305,10 +308,15 @@ argsize = mbstowcs(NULL, arg, 0); #endif if (argsize != (size_t)-1) { - res = (wchar_t *)PyMem_RawMalloc((argsize+1)*sizeof(wchar_t)); + if (argsize == PY_SSIZE_T_MAX) + goto oom; + argsize += 1; + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + goto oom; + res = (wchar_t *)PyMem_RawMalloc(argsize*sizeof(wchar_t)); if (!res) goto oom; - count = mbstowcs(res, arg, argsize+1); + count = mbstowcs(res, arg, argsize); if (count != (size_t)-1) { wchar_t *tmp; /* Only use the result if it contains no @@ -331,6 +339,8 @@ /* Overallocate; as multi-byte characters are in the argument, the actual output could use less memory. */ argsize = strlen(arg) + 1; + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + goto oom; res = (wchar_t*)PyMem_RawMalloc(argsize*sizeof(wchar_t)); if (!res) goto oom; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 23:07:07 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 04 Jan 2015 22:07:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E2_=28closes_=2323165=29?= Message-ID: <20150104220625.11565.88480@psf.io> https://hg.python.org/cpython/rev/d1af6f3a8ce3 changeset: 94019:d1af6f3a8ce3 branch: 3.3 parent: 93996:3b202cc79a38 parent: 94018:1ce98e85929d user: Benjamin Peterson date: Sun Jan 04 16:03:59 2015 -0600 summary: merge 3.2 (closes #23165) files: Misc/NEWS | 3 +++ Python/fileutils.c | 16 +++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ - Issue #22518: Fix integer overflow issues in latin-1 encoding. +- Issue #23165: Perform overflow checks before allocating memory in the + _Py_char2wchar function. + Library ------- diff --git a/Python/fileutils.c b/Python/fileutils.c --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -201,8 +201,11 @@ wchar_t *res; unsigned char *in; wchar_t *out; + size_t argsize = strlen(arg) + 1; - res = PyMem_Malloc((strlen(arg)+1)*sizeof(wchar_t)); + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + return NULL; + res = PyMem_Malloc(argsize*sizeof(wchar_t)); if (!res) return NULL; @@ -284,10 +287,15 @@ argsize = mbstowcs(NULL, arg, 0); #endif if (argsize != (size_t)-1) { - res = (wchar_t *)PyMem_Malloc((argsize+1)*sizeof(wchar_t)); + if (argsize == PY_SSIZE_T_MAX) + goto oom; + argsize += 1; + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + goto oom; + res = (wchar_t *)PyMem_Malloc(argsize*sizeof(wchar_t)); if (!res) goto oom; - count = mbstowcs(res, arg, argsize+1); + count = mbstowcs(res, arg, argsize); if (count != (size_t)-1) { wchar_t *tmp; /* Only use the result if it contains no @@ -310,6 +318,8 @@ /* Overallocate; as multi-byte characters are in the argument, the actual output could use less memory. */ argsize = strlen(arg) + 1; + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + goto oom; res = (wchar_t*)PyMem_Malloc(argsize*sizeof(wchar_t)); if (!res) goto oom; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 23:07:07 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 04 Jan 2015 22:07:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_merge_3=2E3_=28closes_=2323165=29?= Message-ID: <20150104220625.72551.16592@psf.io> https://hg.python.org/cpython/rev/d45e16b1ed86 changeset: 94020:d45e16b1ed86 branch: 3.4 parent: 94014:badb7e319ed0 parent: 94019:d1af6f3a8ce3 user: Benjamin Peterson date: Sun Jan 04 16:05:39 2015 -0600 summary: merge 3.3 (closes #23165) files: Misc/NEWS | 3 +++ Python/fileutils.c | 16 +++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,9 @@ - Issue #22518: Fix integer overflow issues in latin-1 encoding. +- Issue #23165: Perform overflow checks before allocating memory in the + _Py_char2wchar function. + Library ------- diff --git a/Python/fileutils.c b/Python/fileutils.c --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -220,8 +220,11 @@ wchar_t *res; unsigned char *in; wchar_t *out; + size_t argsize = strlen(arg) + 1; - res = PyMem_RawMalloc((strlen(arg)+1)*sizeof(wchar_t)); + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + return NULL; + res = PyMem_RawMalloc(argsize*sizeof(wchar_t)); if (!res) return NULL; @@ -303,10 +306,15 @@ argsize = mbstowcs(NULL, arg, 0); #endif if (argsize != (size_t)-1) { - res = (wchar_t *)PyMem_RawMalloc((argsize+1)*sizeof(wchar_t)); + if (argsize == PY_SSIZE_T_MAX) + goto oom; + argsize += 1; + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + goto oom; + res = (wchar_t *)PyMem_RawMalloc(argsize*sizeof(wchar_t)); if (!res) goto oom; - count = mbstowcs(res, arg, argsize+1); + count = mbstowcs(res, arg, argsize); if (count != (size_t)-1) { wchar_t *tmp; /* Only use the result if it contains no @@ -329,6 +337,8 @@ /* Overallocate; as multi-byte characters are in the argument, the actual output could use less memory. */ argsize = strlen(arg) + 1; + if (argsize > PY_SSIZE_T_MAX/sizeof(wchar_t)) + goto oom; res = (wchar_t*)PyMem_RawMalloc(argsize*sizeof(wchar_t)); if (!res) goto oom; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 23:32:01 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 04 Jan 2015 22:32:01 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMxNjcp?= Message-ID: <20150104223023.11591.14734@psf.io> https://hg.python.org/cpython/rev/826831a9a376 changeset: 94023:826831a9a376 parent: 94021:8c4fb312e15d parent: 94022:8ac23d3242b4 user: Benjamin Peterson date: Sun Jan 04 16:30:13 2015 -0600 summary: merge 3.4 (#23167) files: Doc/library/marshal.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/marshal.rst b/Doc/library/marshal.rst --- a/Doc/library/marshal.rst +++ b/Doc/library/marshal.rst @@ -106,7 +106,7 @@ format, version 1 shares interned strings and version 2 uses a binary format for floating point numbers. Version 3 adds support for object instancing and recursion. - The current version is 3. + The current version is 4. .. rubric:: Footnotes -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 4 23:32:01 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 04 Jan 2015 22:32:01 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_the_current_ma?= =?utf-8?q?rshal_version_is_4_=28closes_=2323167=29?= Message-ID: <20150104223022.11575.21099@psf.io> https://hg.python.org/cpython/rev/8ac23d3242b4 changeset: 94022:8ac23d3242b4 branch: 3.4 parent: 94020:d45e16b1ed86 user: Benjamin Peterson date: Sun Jan 04 16:29:48 2015 -0600 summary: the current marshal version is 4 (closes #23167) Patch by Dmitry Kazakov. files: Doc/library/marshal.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/marshal.rst b/Doc/library/marshal.rst --- a/Doc/library/marshal.rst +++ b/Doc/library/marshal.rst @@ -106,7 +106,7 @@ format, version 1 shares interned strings and version 2 uses a binary format for floating point numbers. Version 3 adds support for object instancing and recursion. - The current version is 3. + The current version is 4. .. rubric:: Footnotes -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 5 08:19:21 2015 From: python-checkins at python.org (berker.peksag) Date: Mon, 05 Jan 2015 07:19:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2318644=3A_Fix_a_ResourceWarning_in_formatter=2Et?= =?utf-8?b?ZXN0KCku?= Message-ID: <20150105071805.22395.71925@psf.io> https://hg.python.org/cpython/rev/f374e4e6d04b changeset: 94025:f374e4e6d04b parent: 94023:826831a9a376 parent: 94024:f859a61f5853 user: Berker Peksag date: Mon Jan 05 09:20:07 2015 +0200 summary: Issue #18644: Fix a ResourceWarning in formatter.test(). Patch by Vajrasky Kok. files: Lib/formatter.py | 14 +++++++++----- 1 files changed, 9 insertions(+), 5 deletions(-) diff --git a/Lib/formatter.py b/Lib/formatter.py --- a/Lib/formatter.py +++ b/Lib/formatter.py @@ -436,11 +436,15 @@ fp = open(sys.argv[1]) else: fp = sys.stdin - for line in fp: - if line == '\n': - f.end_paragraph(1) - else: - f.add_flowing_data(line) + try: + for line in fp: + if line == '\n': + f.end_paragraph(1) + else: + f.add_flowing_data(line) + finally: + if fp is not sys.stdin: + fp.close() f.end_paragraph(0) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 5 08:19:21 2015 From: python-checkins at python.org (berker.peksag) Date: Mon, 05 Jan 2015 07:19:21 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE4NjQ0?= =?utf-8?q?=3A_Fix_a_ResourceWarning_in_formatter=2Etest=28=29=2E?= Message-ID: <20150105071804.8761.47447@psf.io> https://hg.python.org/cpython/rev/f859a61f5853 changeset: 94024:f859a61f5853 branch: 3.4 parent: 94022:8ac23d3242b4 user: Berker Peksag date: Mon Jan 05 09:19:40 2015 +0200 summary: Issue #18644: Fix a ResourceWarning in formatter.test(). Patch by Vajrasky Kok. files: Lib/formatter.py | 14 +++++++++----- 1 files changed, 9 insertions(+), 5 deletions(-) diff --git a/Lib/formatter.py b/Lib/formatter.py --- a/Lib/formatter.py +++ b/Lib/formatter.py @@ -436,11 +436,15 @@ fp = open(sys.argv[1]) else: fp = sys.stdin - for line in fp: - if line == '\n': - f.end_paragraph(1) - else: - f.add_flowing_data(line) + try: + for line in fp: + if line == '\n': + f.end_paragraph(1) + else: + f.add_flowing_data(line) + finally: + if fp is not sys.stdin: + fp.close() f.end_paragraph(0) -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Mon Jan 5 09:11:30 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 05 Jan 2015 09:11:30 +0100 Subject: [Python-checkins] Daily reference leaks (826831a9a376): sum=6 Message-ID: results for 826831a9a376 on branch "default" -------------------------------------------- test_asyncio leaked [0, 0, 3] memory blocks, sum=3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogqFVfLC', '-x'] From python-checkins at python.org Mon Jan 5 10:06:45 2015 From: python-checkins at python.org (ned.deily) Date: Mon, 05 Jan 2015 09:06:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322165=3A_merge_from_3=2E4?= Message-ID: <20150105090607.125884.47082@psf.io> https://hg.python.org/cpython/rev/85258e08b69b changeset: 94027:85258e08b69b parent: 94025:f374e4e6d04b parent: 94026:1bc41bbbe02d user: Ned Deily date: Mon Jan 05 01:05:36 2015 -0800 summary: Issue #22165: merge from 3.4 files: Lib/test/test_httpservers.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py --- a/Lib/test/test_httpservers.py +++ b/Lib/test/test_httpservers.py @@ -269,6 +269,7 @@ self.assertEqual(data, body) return body + @support.requires_mac_ver(10, 5) @unittest.skipUnless(support.TESTFN_UNDECODABLE, 'need support.TESTFN_UNDECODABLE') def test_undecodable_filename(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 5 10:06:45 2015 From: python-checkins at python.org (ned.deily) Date: Mon, 05 Jan 2015 09:06:45 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyMTY1?= =?utf-8?q?=3A_Skip_test=5Fundecodable=5Ffilename_on_OS_X_prior_to_10=2E5?= =?utf-8?q?=2E?= Message-ID: <20150105090606.22417.1634@psf.io> https://hg.python.org/cpython/rev/1bc41bbbe02d changeset: 94026:1bc41bbbe02d branch: 3.4 parent: 94024:f859a61f5853 user: Ned Deily date: Mon Jan 05 01:02:30 2015 -0800 summary: Issue #22165: Skip test_undecodable_filename on OS X prior to 10.5. 10.4 systems do not allow creation of files with such filenames. files: Lib/test/test_httpservers.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py --- a/Lib/test/test_httpservers.py +++ b/Lib/test/test_httpservers.py @@ -269,6 +269,7 @@ self.assertEqual(data, body) return body + @support.requires_mac_ver(10, 5) @unittest.skipUnless(support.TESTFN_UNDECODABLE, 'need support.TESTFN_UNDECODABLE') def test_undecodable_filename(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 5 21:39:16 2015 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 05 Jan 2015 20:39:16 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <20150105203912.8751.41009@psf.io> https://hg.python.org/cpython/rev/a12da014d394 changeset: 94030:a12da014d394 parent: 94027:85258e08b69b parent: 94029:57a8befe75d4 user: Benjamin Peterson date: Mon Jan 05 14:39:06 2015 -0600 summary: merge 3.4 files: Doc/extending/extending.rst | 15 ++++++++++----- 1 files changed, 10 insertions(+), 5 deletions(-) diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -20,12 +20,17 @@ The compilation of an extension module depends on its intended use as well as on your system setup; details are given in later chapters. -Do note that if your use case is calling C library functions or system calls, -you should consider using the :mod:`ctypes` module rather than writing custom -C code. Not only does :mod:`ctypes` let you write Python code to interface -with C code, but it is more portable between implementations of Python than -writing and compiling an extension module which typically ties you to CPython. +.. note:: + The C extension interface is specific to CPython, and extension modules do + not work on other Python implementations. In many cases, it is possible to + avoid writing C extensions and preserve portability to other implementations. + For example, if your use case is calling C library functions or system calls, + you should consider using the :mod:`ctypes` module or the `cffi + `_ library rather than writing custom C code. + These modules let you write Python code to interface with C code and are more + portable between implementations of Python than writing and compiling a C + extension module. .. _extending-simpleexample: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 5 21:39:16 2015 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 05 Jan 2015 20:39:16 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_emphasize_that?= =?utf-8?q?_cffi_is_better_than_extension_modules_for_portability?= Message-ID: <20150105203912.72567.41320@psf.io> https://hg.python.org/cpython/rev/57a8befe75d4 changeset: 94029:57a8befe75d4 branch: 3.4 parent: 94026:1bc41bbbe02d user: Benjamin Peterson date: Mon Jan 05 14:38:46 2015 -0600 summary: emphasize that cffi is better than extension modules for portability files: Doc/extending/extending.rst | 15 ++++++++++----- 1 files changed, 10 insertions(+), 5 deletions(-) diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -20,12 +20,17 @@ The compilation of an extension module depends on its intended use as well as on your system setup; details are given in later chapters. -Do note that if your use case is calling C library functions or system calls, -you should consider using the :mod:`ctypes` module rather than writing custom -C code. Not only does :mod:`ctypes` let you write Python code to interface -with C code, but it is more portable between implementations of Python than -writing and compiling an extension module which typically ties you to CPython. +.. note:: + The C extension interface is specific to CPython, and extension modules do + not work on other Python implementations. In many cases, it is possible to + avoid writing C extensions and preserve portability to other implementations. + For example, if your use case is calling C library functions or system calls, + you should consider using the :mod:`ctypes` module or the `cffi + `_ library rather than writing custom C code. + These modules let you write Python code to interface with C code and are more + portable between implementations of Python than writing and compiling a C + extension module. .. _extending-simpleexample: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 5 21:39:16 2015 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 05 Jan 2015 20:39:16 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_emphasize_that?= =?utf-8?q?_cffi_is_better_than_extension_modules_for_portability?= Message-ID: <20150105203912.125894.46618@psf.io> https://hg.python.org/cpython/rev/518f40815684 changeset: 94028:518f40815684 branch: 2.7 parent: 94017:e8342b3154d1 user: Benjamin Peterson date: Mon Jan 05 14:38:46 2015 -0600 summary: emphasize that cffi is better than extension modules for portability files: Doc/extending/extending.rst | 15 ++++++++++----- 1 files changed, 10 insertions(+), 5 deletions(-) diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -20,12 +20,17 @@ The compilation of an extension module depends on its intended use as well as on your system setup; details are given in later chapters. -Do note that if your use case is calling C library functions or system calls, -you should consider using the :mod:`ctypes` module rather than writing custom -C code. Not only does :mod:`ctypes` let you write Python code to interface -with C code, but it is more portable between implementations of Python than -writing and compiling an extension module which typically ties you to CPython. +.. note:: + The C extension interface is specific to CPython, and extension modules do + not work on other Python implementations. In many cases, it is possible to + avoid writing C extensions and preserve portability to other implementations. + For example, if your use case is calling C library functions or system calls, + you should consider using the :mod:`ctypes` module or the `cffi + `_ library rather than writing custom C code. + These modules let you write Python code to interface with C code and are more + portable between implementations of Python than writing and compiling a C + extension module. .. _extending-simpleexample: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 01:05:48 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 00:05:48 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDQ2?= =?utf-8?q?=3A_Expose_the_BaseEventLoop_class_in_the_asyncio_namespace?= Message-ID: <20150106000541.22395.88746@psf.io> https://hg.python.org/cpython/rev/ddf6b78faed9 changeset: 94031:ddf6b78faed9 branch: 3.4 parent: 94029:57a8befe75d4 user: Victor Stinner date: Tue Jan 06 01:03:58 2015 +0100 summary: Issue #23046: Expose the BaseEventLoop class in the asyncio namespace files: Lib/asyncio/__init__.py | 4 +++- Lib/asyncio/base_events.py | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/__init__.py b/Lib/asyncio/__init__.py --- a/Lib/asyncio/__init__.py +++ b/Lib/asyncio/__init__.py @@ -18,6 +18,7 @@ import _overlapped # Will also be exported. # This relies on each of the submodules having an __all__ variable. +from .base_events import * from .coroutines import * from .events import * from .futures import * @@ -29,7 +30,8 @@ from .tasks import * from .transports import * -__all__ = (coroutines.__all__ + +__all__ = (base_events.__all__ + + coroutines.__all__ + events.__all__ + futures.__all__ + locks.__all__ + diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -35,7 +35,7 @@ from .log import logger -__all__ = ['BaseEventLoop', 'Server'] +__all__ = ['BaseEventLoop'] # Argument for default thread pool executor creation. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 01:05:48 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 00:05:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150106000541.72577.94329@psf.io> https://hg.python.org/cpython/rev/5c5f1fe559d2 changeset: 94032:5c5f1fe559d2 parent: 94030:a12da014d394 parent: 94031:ddf6b78faed9 user: Victor Stinner date: Tue Jan 06 01:04:38 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/__init__.py | 4 +++- Lib/asyncio/base_events.py | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/__init__.py b/Lib/asyncio/__init__.py --- a/Lib/asyncio/__init__.py +++ b/Lib/asyncio/__init__.py @@ -18,6 +18,7 @@ import _overlapped # Will also be exported. # This relies on each of the submodules having an __all__ variable. +from .base_events import * from .coroutines import * from .events import * from .futures import * @@ -29,7 +30,8 @@ from .tasks import * from .transports import * -__all__ = (coroutines.__all__ + +__all__ = (base_events.__all__ + + coroutines.__all__ + events.__all__ + futures.__all__ + locks.__all__ + diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -35,7 +35,7 @@ from .log import logger -__all__ = ['BaseEventLoop', 'Server'] +__all__ = ['BaseEventLoop'] # Argument for default thread pool executor creation. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 01:15:03 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 00:15:03 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTQw?= =?utf-8?q?=2C_asyncio=3A_Fix_cancellation_of_Process=2Ewait=28=29=2E_Chec?= =?utf-8?q?k_the_state_of?= Message-ID: <20150106001452.11577.72439@psf.io> https://hg.python.org/cpython/rev/7c9b9d2514bb changeset: 94033:7c9b9d2514bb branch: 3.4 parent: 94031:ddf6b78faed9 user: Victor Stinner date: Tue Jan 06 01:13:49 2015 +0100 summary: Issue #23140, asyncio: Fix cancellation of Process.wait(). Check the state of the waiter future before setting its result. files: Lib/asyncio/subprocess.py | 3 +- Lib/test/test_asyncio/test_subprocess.py | 28 ++++++++++++ 2 files changed, 30 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -96,7 +96,8 @@ returncode = self._transport.get_returncode() while self._waiters: waiter = self._waiters.popleft() - waiter.set_result(returncode) + if not waiter.cancelled(): + waiter.set_result(returncode) class Process: diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -223,6 +223,34 @@ self.assertEqual(output.rstrip(), b'3') self.assertEqual(exitcode, 0) + def test_cancel_process_wait(self): + # Issue #23140: cancel Process.wait() + + @asyncio.coroutine + def wait_proc(proc, event): + event.set() + yield from proc.wait() + + @asyncio.coroutine + def cancel_wait(): + proc = yield from asyncio.create_subprocess_exec( + *PROGRAM_BLOCKED, + loop=self.loop) + + # Create an internal future waiting on the process exit + event = asyncio.Event(loop=self.loop) + task = self.loop.create_task(wait_proc(proc, event)) + yield from event.wait() + + # Cancel the future + task.cancel() + + # Kill the process and wait until it is done + proc.kill() + yield from proc.wait() + + self.loop.run_until_complete(cancel_wait()) + if sys.platform != 'win32': # Unix -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 01:15:03 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 00:15:03 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150106001452.11581.13067@psf.io> https://hg.python.org/cpython/rev/454e04786fcc changeset: 94034:454e04786fcc parent: 94032:5c5f1fe559d2 parent: 94033:7c9b9d2514bb user: Victor Stinner date: Tue Jan 06 01:14:09 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/subprocess.py | 3 +- Lib/test/test_asyncio/test_subprocess.py | 28 ++++++++++++ 2 files changed, 30 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -96,7 +96,8 @@ returncode = self._transport.get_returncode() while self._waiters: waiter = self._waiters.popleft() - waiter.set_result(returncode) + if not waiter.cancelled(): + waiter.set_result(returncode) class Process: diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -223,6 +223,34 @@ self.assertEqual(output.rstrip(), b'3') self.assertEqual(exitcode, 0) + def test_cancel_process_wait(self): + # Issue #23140: cancel Process.wait() + + @asyncio.coroutine + def wait_proc(proc, event): + event.set() + yield from proc.wait() + + @asyncio.coroutine + def cancel_wait(): + proc = yield from asyncio.create_subprocess_exec( + *PROGRAM_BLOCKED, + loop=self.loop) + + # Create an internal future waiting on the process exit + event = asyncio.Event(loop=self.loop) + task = self.loop.create_task(wait_proc(proc, event)) + yield from event.wait() + + # Cancel the future + task.cancel() + + # Kill the process and wait until it is done + proc.kill() + yield from proc.wait() + + self.loop.run_until_complete(cancel_wait()) + if sys.platform != 'win32': # Unix -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 01:23:55 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 00:23:55 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150106002348.72555.80224@psf.io> https://hg.python.org/cpython/rev/bfd35f76fd0b changeset: 94036:bfd35f76fd0b parent: 94034:454e04786fcc parent: 94035:990ce80d8283 user: Victor Stinner date: Tue Jan 06 01:22:54 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/test/test_asyncio/test_subprocess.py | 14 +++++------ 1 files changed, 6 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -227,20 +227,18 @@ # Issue #23140: cancel Process.wait() @asyncio.coroutine - def wait_proc(proc, event): - event.set() - yield from proc.wait() - - @asyncio.coroutine def cancel_wait(): proc = yield from asyncio.create_subprocess_exec( *PROGRAM_BLOCKED, loop=self.loop) # Create an internal future waiting on the process exit - event = asyncio.Event(loop=self.loop) - task = self.loop.create_task(wait_proc(proc, event)) - yield from event.wait() + task = self.loop.create_task(proc.wait()) + self.loop.call_soon(task.cancel) + try: + yield from task + except asyncio.CancelledError: + pass # Cancel the future task.cancel() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 01:23:56 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 00:23:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTQw?= =?utf-8?q?=2C_asyncio=3A_Simplify_the_unit_test?= Message-ID: <20150106002348.22417.89847@psf.io> https://hg.python.org/cpython/rev/990ce80d8283 changeset: 94035:990ce80d8283 branch: 3.4 parent: 94033:7c9b9d2514bb user: Victor Stinner date: Tue Jan 06 01:22:45 2015 +0100 summary: Issue #23140, asyncio: Simplify the unit test files: Lib/test/test_asyncio/test_subprocess.py | 14 +++++------ 1 files changed, 6 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -227,20 +227,18 @@ # Issue #23140: cancel Process.wait() @asyncio.coroutine - def wait_proc(proc, event): - event.set() - yield from proc.wait() - - @asyncio.coroutine def cancel_wait(): proc = yield from asyncio.create_subprocess_exec( *PROGRAM_BLOCKED, loop=self.loop) # Create an internal future waiting on the process exit - event = asyncio.Event(loop=self.loop) - task = self.loop.create_task(wait_proc(proc, event)) - yield from event.wait() + task = self.loop.create_task(proc.wait()) + self.loop.call_soon(task.cancel) + try: + yield from task + except asyncio.CancelledError: + pass # Cancel the future task.cancel() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 07:00:17 2015 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 06 Jan 2015 06:00:17 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <20150106060015.11581.96544@psf.io> https://hg.python.org/cpython/rev/141f6a3c4153 changeset: 94038:141f6a3c4153 parent: 94036:bfd35f76fd0b parent: 94037:09b0da38ce8d user: Raymond Hettinger date: Mon Jan 05 22:00:08 2015 -0800 summary: merge files: Lib/functools.py | 153 ++++++++++++++++++---------------- 1 files changed, 79 insertions(+), 74 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -89,110 +89,115 @@ ### total_ordering class decorator ################################################################################ -# The correct way to indicate that a comparison operation doesn't -# recognise the other type is to return NotImplemented and let the -# interpreter handle raising TypeError if both operands return -# NotImplemented from their respective comparison methods -# -# This makes the implementation of total_ordering more complicated, since -# we need to be careful not to trigger infinite recursion when two -# different types that both use this decorator encounter each other. -# -# For example, if a type implements __lt__, it's natural to define -# __gt__ as something like: -# -# lambda self, other: not self < other and not self == other -# -# However, using the operator syntax like that ends up invoking the full -# type checking machinery again and means we can end up bouncing back and -# forth between the two operands until we run out of stack space. -# -# The solution is to define helper functions that invoke the appropriate -# magic methods directly, ensuring we only try each operand once, and -# return NotImplemented immediately if it is returned from the -# underlying user provided method. Using this scheme, the __gt__ derived -# from a user provided __lt__ becomes: -# -# 'def __gt__(self, other):' + _not_op_and_not_eq % '__lt__' +# The total ordering functions all invoke the root magic method directly +# rather than using the corresponding operator. This avoids possible +# infinite recursion that could occur when the operator dispatch logic +# detects a NotImplemented result and then calls a reflected method. -# "not a < b" handles "a >= b" -# "not a <= b" handles "a > b" -# "not a >= b" handles "a < b" -# "not a > b" handles "a <= b" -_not_op = ''' - op_result = self.%s(other) +def _gt_from_lt(self, other): + 'Return a > b. Computed by @total_ordering from (not a < b) and (a != b).' + op_result = self.__lt__(other) + if op_result is NotImplemented: + return NotImplemented + return not op_result and self != other + +def _le_from_lt(self, other): + 'Return a <= b. Computed by @total_ordering from (a < b) or (a == b).' + op_result = self.__lt__(other) + return op_result or self == other + +def _ge_from_lt(self, other): + 'Return a >= b. Computed by @total_ordering from (not a < b).' + op_result = self.__lt__(other) if op_result is NotImplemented: return NotImplemented return not op_result -''' -# "a > b or a == b" handles "a >= b" -# "a < b or a == b" handles "a <= b" -_op_or_eq = ''' - op_result = self.%s(other) +def _ge_from_le(self, other): + 'Return a >= b. Computed by @total_ordering from (not a <= b) or (a == b).' + op_result = self.__le__(other) if op_result is NotImplemented: return NotImplemented - return op_result or self == other -''' + return not op_result or self == other -# "not (a < b or a == b)" handles "a > b" -# "not a < b and a != b" is equivalent -# "not (a > b or a == b)" handles "a < b" -# "not a > b and a != b" is equivalent -_not_op_and_not_eq = ''' - op_result = self.%s(other) +def _lt_from_le(self, other): + 'Return a < b. Computed by @total_ordering from (a <= b) and (a != b).' + op_result = self.__le__(other) + if op_result is NotImplemented: + return NotImplemented + return op_result and self != other + +def _gt_from_le(self, other): + 'Return a > b. Computed by @total_ordering from (not a <= b).' + op_result = self.__le__(other) + if op_result is NotImplemented: + return NotImplemented + return not op_result + +def _lt_from_gt(self, other): + 'Return a < b. Computed by @total_ordering from (not a > b) and (a != b).' + op_result = self.__gt__(other) if op_result is NotImplemented: return NotImplemented return not op_result and self != other -''' -# "not a <= b or a == b" handles "a >= b" -# "not a >= b or a == b" handles "a <= b" -_not_op_or_eq = ''' - op_result = self.%s(other) +def _ge_from_gt(self, other): + 'Return a >= b. Computed by @total_ordering from (a > b) or (a == b).' + op_result = self.__gt__(other) + return op_result or self == other + +def _le_from_gt(self, other): + 'Return a <= b. Computed by @total_ordering from (not a > b).' + op_result = self.__gt__(other) + if op_result is NotImplemented: + return NotImplemented + return not op_result + +def _le_from_ge(self, other): + 'Return a <= b. Computed by @total_ordering from (not a >= b) or (a == b).' + op_result = self.__ge__(other) if op_result is NotImplemented: return NotImplemented return not op_result or self == other -''' -# "a <= b and not a == b" handles "a < b" -# "a >= b and not a == b" handles "a > b" -_op_and_not_eq = ''' - op_result = self.%s(other) +def _gt_from_ge(self, other): + 'Return a > b. Computed by @total_ordering from (a >= b) and (a != b).' + op_result = self.__ge__(other) if op_result is NotImplemented: return NotImplemented return op_result and self != other -''' + +def _lt_from_ge(self, other): + 'Return a < b. Computed by @total_ordering from (not a >= b).' + op_result = self.__ge__(other) + if op_result is NotImplemented: + return NotImplemented + return not op_result def total_ordering(cls): """Class decorator that fills in missing ordering methods""" convert = { - '__lt__': {'__gt__': _not_op_and_not_eq, - '__le__': _op_or_eq, - '__ge__': _not_op}, - '__le__': {'__ge__': _not_op_or_eq, - '__lt__': _op_and_not_eq, - '__gt__': _not_op}, - '__gt__': {'__lt__': _not_op_and_not_eq, - '__ge__': _op_or_eq, - '__le__': _not_op}, - '__ge__': {'__le__': _not_op_or_eq, - '__gt__': _op_and_not_eq, - '__lt__': _not_op} + '__lt__': [('__gt__', _gt_from_lt), + ('__le__', _le_from_lt), + ('__ge__', _ge_from_lt)], + '__le__': [('__ge__', _ge_from_le), + ('__lt__', _lt_from_le), + ('__gt__', _gt_from_le)], + '__gt__': [('__lt__', _lt_from_gt), + ('__ge__', _ge_from_gt), + ('__le__', _le_from_gt)], + '__ge__': [('__le__', _le_from_ge), + ('__gt__', _gt_from_ge), + ('__lt__', _lt_from_ge)] } # Find user-defined comparisons (not those inherited from object). roots = [op for op in convert if getattr(cls, op, None) is not getattr(object, op, None)] if not roots: raise ValueError('must define at least one ordering operation: < > <= >=') root = max(roots) # prefer __lt__ to __le__ to __gt__ to __ge__ - for opname, opfunc in convert[root].items(): + for opname, opfunc in convert[root]: if opname not in roots: - namespace = {} - exec('def %s(self, other):%s' % (opname, opfunc % root), namespace) - opfunc = namespace[opname] - opfunc.__qualname__ = '%s.%s' % (cls.__qualname__, opname) - opfunc.__module__ = cls.__module__ - opfunc.__doc__ = getattr(int, opname).__doc__ + opfunc.__name__ = opname setattr(cls, opname, opfunc) return cls -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 07:00:17 2015 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 06 Jan 2015 06:00:17 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTMy?= =?utf-8?q?=3A_Mitigate_regression_in_speed_and_clarity_in?= Message-ID: <20150106060015.125896.92510@psf.io> https://hg.python.org/cpython/rev/09b0da38ce8d changeset: 94037:09b0da38ce8d branch: 3.4 parent: 94035:990ce80d8283 user: Raymond Hettinger date: Mon Jan 05 21:52:10 2015 -0800 summary: Issue #23132: Mitigate regression in speed and clarity in functools.total_ordering. files: Lib/functools.py | 140 +++++++++++++++++++--------------- Misc/NEWS | 2 + 2 files changed, 79 insertions(+), 63 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -89,91 +89,106 @@ ### total_ordering class decorator ################################################################################ -# The correct way to indicate that a comparison operation doesn't -# recognise the other type is to return NotImplemented and let the -# interpreter handle raising TypeError if both operands return -# NotImplemented from their respective comparison methods -# -# This makes the implementation of total_ordering more complicated, since -# we need to be careful not to trigger infinite recursion when two -# different types that both use this decorator encounter each other. -# -# For example, if a type implements __lt__, it's natural to define -# __gt__ as something like: -# -# lambda self, other: not self < other and not self == other -# -# However, using the operator syntax like that ends up invoking the full -# type checking machinery again and means we can end up bouncing back and -# forth between the two operands until we run out of stack space. -# -# The solution is to define helper functions that invoke the appropriate -# magic methods directly, ensuring we only try each operand once, and -# return NotImplemented immediately if it is returned from the -# underlying user provided method. Using this scheme, the __gt__ derived -# from a user provided __lt__ becomes: -# -# lambda self, other: _not_op_and_not_eq(self.__lt__, self, other)) +# The total ordering functions all invoke the root magic method directly +# rather than using the corresponding operator. This avoids possible +# infinite recursion that could occur when the operator dispatch logic +# detects a NotImplemented result and then calls a reflected method. -def _not_op(op, other): - # "not a < b" handles "a >= b" - # "not a <= b" handles "a > b" - # "not a >= b" handles "a < b" - # "not a > b" handles "a <= b" - op_result = op(other) +def _gt_from_lt(self, other): + 'Return a > b. Computed by @total_ordering from (not a < b) and (a != b).' + op_result = self.__lt__(other) + if op_result is NotImplemented: + return NotImplemented + return not op_result and self != other + +def _le_from_lt(self, other): + 'Return a <= b. Computed by @total_ordering from (a < b) or (a == b).' + op_result = self.__lt__(other) + return op_result or self == other + +def _ge_from_lt(self, other): + 'Return a >= b. Computed by @total_ordering from (not a < b).' + op_result = self.__lt__(other) if op_result is NotImplemented: return NotImplemented return not op_result -def _op_or_eq(op, self, other): - # "a < b or a == b" handles "a <= b" - # "a > b or a == b" handles "a >= b" - op_result = op(other) +def _ge_from_le(self, other): + 'Return a >= b. Computed by @total_ordering from (not a <= b) or (a == b).' + op_result = self.__le__(other) if op_result is NotImplemented: return NotImplemented - return op_result or self == other + return not op_result or self == other -def _not_op_and_not_eq(op, self, other): - # "not (a < b or a == b)" handles "a > b" - # "not a < b and a != b" is equivalent - # "not (a > b or a == b)" handles "a < b" - # "not a > b and a != b" is equivalent - op_result = op(other) +def _lt_from_le(self, other): + 'Return a < b. Computed by @total_ordering from (a <= b) and (a != b).' + op_result = self.__le__(other) + if op_result is NotImplemented: + return NotImplemented + return op_result and self != other + +def _gt_from_le(self, other): + 'Return a > b. Computed by @total_ordering from (not a <= b).' + op_result = self.__le__(other) + if op_result is NotImplemented: + return NotImplemented + return not op_result + +def _lt_from_gt(self, other): + 'Return a < b. Computed by @total_ordering from (not a > b) and (a != b).' + op_result = self.__gt__(other) if op_result is NotImplemented: return NotImplemented return not op_result and self != other -def _not_op_or_eq(op, self, other): - # "not a <= b or a == b" handles "a >= b" - # "not a >= b or a == b" handles "a <= b" - op_result = op(other) +def _ge_from_gt(self, other): + 'Return a >= b. Computed by @total_ordering from (a > b) or (a == b).' + op_result = self.__gt__(other) + return op_result or self == other + +def _le_from_gt(self, other): + 'Return a <= b. Computed by @total_ordering from (not a > b).' + op_result = self.__gt__(other) + if op_result is NotImplemented: + return NotImplemented + return not op_result + +def _le_from_ge(self, other): + 'Return a <= b. Computed by @total_ordering from (not a >= b) or (a == b).' + op_result = self.__ge__(other) if op_result is NotImplemented: return NotImplemented return not op_result or self == other -def _op_and_not_eq(op, self, other): - # "a <= b and not a == b" handles "a < b" - # "a >= b and not a == b" handles "a > b" - op_result = op(other) +def _gt_from_ge(self, other): + 'Return a > b. Computed by @total_ordering from (a >= b) and (a != b).' + op_result = self.__ge__(other) if op_result is NotImplemented: return NotImplemented return op_result and self != other +def _lt_from_ge(self, other): + 'Return a < b. Computed by @total_ordering from (not a >= b).' + op_result = self.__ge__(other) + if op_result is NotImplemented: + return NotImplemented + return not op_result + def total_ordering(cls): """Class decorator that fills in missing ordering methods""" convert = { - '__lt__': [('__gt__', lambda self, other: _not_op_and_not_eq(self.__lt__, self, other)), - ('__le__', lambda self, other: _op_or_eq(self.__lt__, self, other)), - ('__ge__', lambda self, other: _not_op(self.__lt__, other))], - '__le__': [('__ge__', lambda self, other: _not_op_or_eq(self.__le__, self, other)), - ('__lt__', lambda self, other: _op_and_not_eq(self.__le__, self, other)), - ('__gt__', lambda self, other: _not_op(self.__le__, other))], - '__gt__': [('__lt__', lambda self, other: _not_op_and_not_eq(self.__gt__, self, other)), - ('__ge__', lambda self, other: _op_or_eq(self.__gt__, self, other)), - ('__le__', lambda self, other: _not_op(self.__gt__, other))], - '__ge__': [('__le__', lambda self, other: _not_op_or_eq(self.__ge__, self, other)), - ('__gt__', lambda self, other: _op_and_not_eq(self.__ge__, self, other)), - ('__lt__', lambda self, other: _not_op(self.__ge__, other))] + '__lt__': [('__gt__', _gt_from_lt), + ('__le__', _le_from_lt), + ('__ge__', _ge_from_lt)], + '__le__': [('__ge__', _ge_from_le), + ('__lt__', _lt_from_le), + ('__gt__', _gt_from_le)], + '__gt__': [('__lt__', _lt_from_gt), + ('__ge__', _ge_from_gt), + ('__le__', _le_from_gt)], + '__ge__': [('__le__', _le_from_ge), + ('__gt__', _gt_from_ge), + ('__lt__', _lt_from_ge)] } # Find user-defined comparisons (not those inherited from object). roots = [op for op in convert if getattr(cls, op, None) is not getattr(object, op, None)] @@ -183,7 +198,6 @@ for opname, opfunc in convert[root]: if opname not in roots: opfunc.__name__ = opname - opfunc.__doc__ = getattr(int, opname).__doc__ setattr(cls, opname, opfunc) return cls diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -47,6 +47,8 @@ - Issue #23111: In the ftplib, make ssl.PROTOCOL_SSLv23 the default protocol version. +- Issue #23132: Mitigate regression in speed and clarity in functools.total_ordering. + - Issue #22585: On OpenBSD 5.6 and newer, os.urandom() now calls getentropy(), instead of reading /dev/urandom, to get pseudo-random bytes. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 07:47:36 2015 From: python-checkins at python.org (zach.ware) Date: Tue, 06 Jan 2015 06:47:36 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Cosmetic_fixes?= =?utf-8?q?_to_the_=27Develop_with_asyncio=27_page?= Message-ID: <20150106064612.125900.81426@psf.io> https://hg.python.org/cpython/rev/37801e3b82e4 changeset: 94039:37801e3b82e4 branch: 3.4 parent: 94037:09b0da38ce8d user: Zachary Ware date: Tue Jan 06 00:40:43 2015 -0600 summary: Cosmetic fixes to the 'Develop with asyncio' page files: Doc/library/asyncio-dev.rst | 18 ++++++++++-------- 1 files changed, 10 insertions(+), 8 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -71,8 +71,8 @@ .. seealso:: - See the :ref:`Synchronization primitives ` section to - synchronize tasks. + The :ref:`Synchronization primitives ` section describes ways + to synchronize tasks. .. _asyncio-handle-blocking: @@ -112,8 +112,8 @@ ---------------------------------------- When a coroutine function is called and its result is not passed to -:func:`async` or to the :meth:`BaseEventLoop.create_task` method: the execution -of the coroutine objet will never be scheduled and it is probably a bug. +:func:`async` or to the :meth:`BaseEventLoop.create_task` method, the execution +of the coroutine object will never be scheduled which is probably a bug. :ref:`Enable the debug mode of asyncio ` to :ref:`log a warning ` to detect it. @@ -147,7 +147,7 @@ Python usually calls :func:`sys.displayhook` on unhandled exceptions. If :meth:`Future.set_exception` is called, but the exception is never consumed, -:func:`sys.displayhook` is not called. Instead, a :ref:`a log is emitted +:func:`sys.displayhook` is not called. Instead, :ref:`a log is emitted ` when the future is deleted by the garbage collector, with the traceback where the exception was raised. @@ -195,7 +195,7 @@ raise Exception("not consumed") Exception: not consumed -There are different options to fix this issue. The first option is to chain to +There are different options to fix this issue. The first option is to chain the coroutine in another coroutine and use classic try/except:: @asyncio.coroutine @@ -218,10 +218,12 @@ except Exception: print("exception consumed") -See also the :meth:`Future.exception` method. +.. seealso:: + The :meth:`Future.exception` method. -Chain correctly coroutines + +Chain coroutines correctly -------------------------- When a coroutine function calls other coroutine functions and tasks, they -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 07:47:36 2015 From: python-checkins at python.org (zach.ware) Date: Tue, 06 Jan 2015 06:47:36 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <20150106064612.8761.1508@psf.io> https://hg.python.org/cpython/rev/8120043810af changeset: 94040:8120043810af parent: 94038:141f6a3c4153 parent: 94039:37801e3b82e4 user: Zachary Ware date: Tue Jan 06 00:45:52 2015 -0600 summary: Merge with 3.4 files: Doc/library/asyncio-dev.rst | 18 ++++++++++-------- 1 files changed, 10 insertions(+), 8 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -71,8 +71,8 @@ .. seealso:: - See the :ref:`Synchronization primitives ` section to - synchronize tasks. + The :ref:`Synchronization primitives ` section describes ways + to synchronize tasks. .. _asyncio-handle-blocking: @@ -112,8 +112,8 @@ ---------------------------------------- When a coroutine function is called and its result is not passed to -:func:`async` or to the :meth:`BaseEventLoop.create_task` method: the execution -of the coroutine objet will never be scheduled and it is probably a bug. +:func:`async` or to the :meth:`BaseEventLoop.create_task` method, the execution +of the coroutine object will never be scheduled which is probably a bug. :ref:`Enable the debug mode of asyncio ` to :ref:`log a warning ` to detect it. @@ -147,7 +147,7 @@ Python usually calls :func:`sys.displayhook` on unhandled exceptions. If :meth:`Future.set_exception` is called, but the exception is never consumed, -:func:`sys.displayhook` is not called. Instead, a :ref:`a log is emitted +:func:`sys.displayhook` is not called. Instead, :ref:`a log is emitted ` when the future is deleted by the garbage collector, with the traceback where the exception was raised. @@ -195,7 +195,7 @@ raise Exception("not consumed") Exception: not consumed -There are different options to fix this issue. The first option is to chain to +There are different options to fix this issue. The first option is to chain the coroutine in another coroutine and use classic try/except:: @asyncio.coroutine @@ -218,10 +218,12 @@ except Exception: print("exception consumed") -See also the :meth:`Future.exception` method. +.. seealso:: + The :meth:`Future.exception` method. -Chain correctly coroutines + +Chain coroutines correctly -------------------------- When a coroutine function calls other coroutine functions and tasks, they -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Tue Jan 6 09:30:19 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 06 Jan 2015 09:30:19 +0100 Subject: [Python-checkins] Daily reference leaks (bfd35f76fd0b): sum=6 Message-ID: results for bfd35f76fd0b on branch "default" -------------------------------------------- test_asyncio leaked [0, 3, 0] memory blocks, sum=3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogjtElM7', '-x'] From python-checkins at python.org Tue Jan 6 11:53:05 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 10:53:05 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_test=5Fssl=3A_add_more_deb?= =?utf-8?q?ug_to_investigate_test=5Fopenssl=5Fversion=28=29_failure_on?= Message-ID: <20150106105203.22395.51727@psf.io> https://hg.python.org/cpython/rev/87976d72fd5c changeset: 94041:87976d72fd5c user: Victor Stinner date: Tue Jan 06 11:51:06 2015 +0100 summary: test_ssl: add more debug to investigate test_openssl_version() failure on OpenBSD with LibreSSL. files: Lib/test/test_ssl.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -312,10 +312,10 @@ # Version string as returned by {Open,Libre}SSL, the format might change if "LibreSSL" in s: self.assertTrue(s.startswith("LibreSSL {:d}.{:d}".format(major, minor)), - (s, t)) + (s, t, hex(n))) else: self.assertTrue(s.startswith("OpenSSL {:d}.{:d}.{:d}".format(major, minor, fix)), - (s, t)) + (s, t, hex(n))) @support.cpython_only def test_refcycle(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 11:56:27 2015 From: python-checkins at python.org (vinay.sajip) Date: Tue, 06 Jan 2015 10:56:27 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2323151=3A_Removed?= =?utf-8?q?_unnecessary_initialization=2E?= Message-ID: <20150106105620.11579.91285@psf.io> https://hg.python.org/cpython/rev/8bfe230db0bc changeset: 94042:8bfe230db0bc user: Vinay Sajip date: Tue Jan 06 10:56:09 2015 +0000 summary: Closes #23151: Removed unnecessary initialization. files: Lib/logging/__init__.py | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1086,7 +1086,6 @@ # # Determine which class to use when instantiating loggers. # -_loggerClass = None def setLoggerClass(klass): """ @@ -1105,7 +1104,6 @@ """ Return the class to be used when instantiating a logger. """ - return _loggerClass class Manager(object): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 12:11:03 2015 From: python-checkins at python.org (vinay.sajip) Date: Tue, 06 Jan 2015 11:11:03 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Clarified_docu?= =?utf-8?q?mentation_for_logging_exception_function/method=2E?= Message-ID: <20150106111100.22413.55383@psf.io> https://hg.python.org/cpython/rev/86374d71d4d2 changeset: 94043:86374d71d4d2 branch: 2.7 parent: 94028:518f40815684 user: Vinay Sajip date: Tue Jan 06 11:10:12 2015 +0000 summary: Clarified documentation for logging exception function/method. files: Doc/library/logging.rst | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -227,8 +227,9 @@ .. method:: Logger.exception(msg, *args, **kwargs) Logs a message with level :const:`ERROR` on this logger. The arguments are - interpreted as for :meth:`debug`. Exception info is added to the logging - message. This method should only be called from an exception handler. + interpreted as for :meth:`debug`, except that any passed *exc_info* is not + inspected. Exception info is always added to the logging message. This method + should only be called from an exception handler. .. method:: Logger.addFilter(filt) @@ -845,8 +846,9 @@ .. function:: exception(msg[, *args[, **kwargs]]) Logs a message with level :const:`ERROR` on the root logger. The arguments are - interpreted as for :func:`debug`. Exception info is added to the logging - message. This function should only be called from an exception handler. + interpreted as for :func:`debug`, except that any passed *exc_info* is not + inspected. Exception info is always added to the logging message. This + function should only be called from an exception handler. .. function:: log(level, msg[, *args[, **kwargs]]) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 12:19:57 2015 From: python-checkins at python.org (vinay.sajip) Date: Tue, 06 Jan 2015 11:19:57 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2321980=3A_Added_a?= =?utf-8?b?IF9fcmVwcl9fIGZvciBMb2dSZWNvcmQu?= Message-ID: <20150106111953.8739.47223@psf.io> https://hg.python.org/cpython/rev/390ffd39631b changeset: 94044:390ffd39631b parent: 94042:8bfe230db0bc user: Vinay Sajip date: Tue Jan 06 11:19:42 2015 +0000 summary: Closes #21980: Added a __repr__ for LogRecord. files: Lib/logging/__init__.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2001-2014 by Vinay Sajip. All Rights Reserved. +# Copyright 2001-2015 by Vinay Sajip. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and its # documentation for any purpose and without fee is hereby granted, @@ -18,7 +18,7 @@ Logging package for Python. Based on PEP 282 and comments thereto in comp.lang.python. -Copyright (C) 2001-2014 Vinay Sajip. All Rights Reserved. +Copyright (C) 2001-2015 Vinay Sajip. All Rights Reserved. To use, simply 'import logging' and log away! """ @@ -316,6 +316,8 @@ return ''%(self.name, self.levelno, self.pathname, self.lineno, self.msg) + __repr__ = __str__ + def getMessage(self): """ Return the message for this LogRecord. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 12:23:51 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 11:23:51 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_python_3=2E4_=28change_already_applied_to_Pyt?= =?utf-8?q?hon_3=2E5=29?= Message-ID: <20150106112350.11577.49205@psf.io> https://hg.python.org/cpython/rev/9654ef862c2b changeset: 94046:9654ef862c2b parent: 94044:390ffd39631b parent: 94045:a8c4925e2359 user: Victor Stinner date: Tue Jan 06 12:23:15 2015 +0100 summary: Null merge python 3.4 (change already applied to Python 3.5) files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 12:23:51 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 11:23:51 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwODk2?= =?utf-8?q?=2C_=2322935=3A_The_ssl=2Eget=5Fserver=5Fcertificate=28=29_func?= =?utf-8?q?tion_now_uses_the?= Message-ID: <20150106112349.72561.51614@psf.io> https://hg.python.org/cpython/rev/a8c4925e2359 changeset: 94045:a8c4925e2359 branch: 3.4 parent: 94039:37801e3b82e4 user: Victor Stinner date: Tue Jan 06 12:21:26 2015 +0100 summary: Issue #20896, #22935: The ssl.get_server_certificate() function now uses the ssl.PROTOCOL_SSLv23 protocol by default, not ssl.PROTOCOL_SSLv3, for maximum compatibility and support platforms where ssl.PROTOCOL_SSLv3 support is disabled. files: Lib/ssl.py | 2 +- Misc/NEWS | 5 +++++ 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -922,7 +922,7 @@ d = pem_cert_string.strip()[len(PEM_HEADER):-len(PEM_FOOTER)] return base64.decodebytes(d.encode('ASCII', 'strict')) -def get_server_certificate(addr, ssl_version=PROTOCOL_SSLv3, ca_certs=None): +def get_server_certificate(addr, ssl_version=PROTOCOL_SSLv23, ca_certs=None): """Retrieve the certificate from the server at the specified address, and return it as a PEM-encoded string. If 'ca_certs' is specified, validate the server cert against it. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,11 @@ Library ------- +- Issue #20896, #22935: The :func:`ssl.get_server_certificate` function now + uses the :data:`~ssl.PROTOCOL_SSLv23` protocol by default, not + :data:`~ssl.PROTOCOL_SSLv3`, for maximum compatibility and support platforms + where :data:`~ssl.PROTOCOL_SSLv3` support is disabled. + - Issue #23111: In the ftplib, make ssl.PROTOCOL_SSLv23 the default protocol version. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 12:40:17 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 11:40:17 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMTY4?= =?utf-8?q?=3A_skip_sys=2Estdin=2Eseek=28=29_test_if_stdin_is_not_a_TTY?= Message-ID: <20150106113957.11575.85700@psf.io> https://hg.python.org/cpython/rev/7f30206d402f changeset: 94047:7f30206d402f branch: 2.7 parent: 94043:86374d71d4d2 user: Victor Stinner date: Tue Jan 06 12:39:45 2015 +0100 summary: Issue #23168: skip sys.stdin.seek() test if stdin is not a TTY files: Lib/test/test_file2k.py | 22 ++++++++++++++-------- 1 files changed, 14 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_file2k.py b/Lib/test/test_file2k.py --- a/Lib/test/test_file2k.py +++ b/Lib/test/test_file2k.py @@ -230,14 +230,20 @@ else: f.close() - def testStdin(self): - # This causes the interpreter to exit on OSF1 v5.1. - if sys.platform != 'osf1V5': - self.assertRaises(IOError, sys.stdin.seek, -1) - else: - print >>sys.__stdout__, ( - ' Skipping sys.stdin.seek(-1), it may crash the interpreter.' - ' Test manually.') + def testStdinSeek(self): + if sys.platform == 'osf1V5': + # This causes the interpreter to exit on OSF1 v5.1. + self.skipTest('Skipping sys.stdin.seek(-1), it may crash ' + 'the interpreter. Test manually.') + + if not sys.stdin.isatty(): + # Issue #23168: if stdin is redirected to a file, stdin becomes + # seekable + self.skipTest('stdin must be a TTY in this test') + + self.assertRaises(IOError, sys.stdin.seek, -1) + + def testStdinTruncate(self): self.assertRaises(IOError, sys.stdin.truncate) def testUnicodeOpen(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 14:00:50 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 13:00:50 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_3=2E4_=28changes_already_applied_to_Python_3?= =?utf-8?b?LjUp?= Message-ID: <20150106130030.22423.41176@psf.io> https://hg.python.org/cpython/rev/535232275fa2 changeset: 94051:535232275fa2 parent: 94048:05d7550bd2d9 parent: 94050:7f82f50fdad0 user: Victor Stinner date: Tue Jan 06 13:58:57 2015 +0100 summary: Null merge 3.4 (changes already applied to Python 3.5) files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 14:00:50 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 13:00:50 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323177=3A_Document?= =?utf-8?q?_that_ssl=2ERAND=5Fegd=28=29_is_not_available_with_LibreSSL?= Message-ID: <20150106130029.72565.36209@psf.io> https://hg.python.org/cpython/rev/05d7550bd2d9 changeset: 94048:05d7550bd2d9 parent: 94046:9654ef862c2b user: Victor Stinner date: Tue Jan 06 13:53:09 2015 +0100 summary: Issue #23177: Document that ssl.RAND_egd() is not available with LibreSSL files: Doc/library/ssl.rst | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -327,6 +327,8 @@ See http://egd.sourceforge.net/ or http://prngd.sourceforge.net/ for sources of entropy-gathering daemons. + Availability: not available with LibreSSL. + .. function:: RAND_add(bytes, entropy) Mixes the given *bytes* into the SSL pseudo-random number generator. The -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 14:00:50 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 13:00:50 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMzU2?= =?utf-8?q?=3A_Make_ssl=2ERAND=5Fegd=28=29_optional_to_support_LibreSSL=2E?= =?utf-8?q?_The?= Message-ID: <20150106130030.8739.13517@psf.io> https://hg.python.org/cpython/rev/eddcb6671a48 changeset: 94049:eddcb6671a48 branch: 2.7 parent: 94047:7f30206d402f user: Victor Stinner date: Tue Jan 06 13:53:37 2015 +0100 summary: Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The availability of the function is checked during the compilation. Patch written by Bernard Spil. files: Doc/library/ssl.rst | 2 + Lib/socket.py | 6 +++- Lib/ssl.py | 7 ++++- Lib/test/test_ssl.py | 5 ++- Misc/NEWS | 4 +++ Modules/_ssl.c | 13 +++++++-- configure | 42 ++++++++++++++++++++++++++++++++ configure.ac | 3 ++ pyconfig.h.in | 3 ++ 9 files changed, 78 insertions(+), 7 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -299,6 +299,8 @@ See http://egd.sourceforge.net/ or http://prngd.sourceforge.net/ for sources of entropy-gathering daemons. + Availability: not available with LibreSSL. + .. function:: RAND_add(bytes, entropy) Mixes the given *bytes* into the SSL pseudo-random number generator. The diff --git a/Lib/socket.py b/Lib/socket.py --- a/Lib/socket.py +++ b/Lib/socket.py @@ -67,7 +67,6 @@ from _ssl import SSLError as sslerror from _ssl import \ RAND_add, \ - RAND_egd, \ RAND_status, \ SSL_ERROR_ZERO_RETURN, \ SSL_ERROR_WANT_READ, \ @@ -78,6 +77,11 @@ SSL_ERROR_WANT_CONNECT, \ SSL_ERROR_EOF, \ SSL_ERROR_INVALID_ERROR_CODE + try: + from _ssl import RAND_egd + except ImportError: + # LibreSSL does not provide RAND_egd + pass import os, sys, warnings diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -106,7 +106,12 @@ from _ssl import (VERIFY_DEFAULT, VERIFY_CRL_CHECK_LEAF, VERIFY_CRL_CHECK_CHAIN, VERIFY_X509_STRICT) from _ssl import txt2obj as _txt2obj, nid2obj as _nid2obj -from _ssl import RAND_status, RAND_egd, RAND_add +from _ssl import RAND_status, RAND_add +try: + from _ssl import RAND_egd +except ImportError: + # LibreSSL does not provide RAND_egd + pass def _import_symbols(prefix): for n in dir(_ssl): diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -169,8 +169,9 @@ sys.stdout.write("\n RAND_status is %d (%s)\n" % (v, (v and "sufficient randomness") or "insufficient randomness")) - self.assertRaises(TypeError, ssl.RAND_egd, 1) - self.assertRaises(TypeError, ssl.RAND_egd, 'foo', 1) + if hasattr(ssl, 'RAND_egd'): + self.assertRaises(TypeError, ssl.RAND_egd, 1) + self.assertRaises(TypeError, ssl.RAND_egd, 'foo', 1) ssl.RAND_add("this is a random string", 75.0) def test_parse_cert(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,10 @@ Library ------- +- Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The + availability of the function is checked during the compilation. Patch written + by Bernard Spil. + - Backport the context argument to ftplib.FTP_TLS. - Issue #23111: Maximize compatibility in protocol versions of ftplib.FTP_TLS. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -3301,6 +3301,11 @@ It is necessary to seed the PRNG with RAND_add() on some platforms before\n\ using the ssl() function."); +#endif /* HAVE_OPENSSL_RAND */ + + +#ifdef HAVE_RAND_EGD + static PyObject * PySSL_RAND_egd(PyObject *self, PyObject *arg) { @@ -3327,7 +3332,7 @@ Returns number of bytes read. Raises SSLError if connection to EGD\n\ fails or if it does not provide enough data to seed PRNG."); -#endif /* HAVE_OPENSSL_RAND */ +#endif /* HAVE_RAND_EGD */ PyDoc_STRVAR(PySSL_get_default_verify_paths_doc, @@ -3720,10 +3725,12 @@ #ifdef HAVE_OPENSSL_RAND {"RAND_add", PySSL_RAND_add, METH_VARARGS, PySSL_RAND_add_doc}, + {"RAND_status", (PyCFunction)PySSL_RAND_status, METH_NOARGS, + PySSL_RAND_status_doc}, +#endif +#ifdef HAVE_RAND_EGD {"RAND_egd", PySSL_RAND_egd, METH_VARARGS, PySSL_RAND_egd_doc}, - {"RAND_status", (PyCFunction)PySSL_RAND_status, METH_NOARGS, - PySSL_RAND_status_doc}, #endif {"get_default_verify_paths", (PyCFunction)PySSL_get_default_verify_paths, METH_NOARGS, PySSL_get_default_verify_paths_doc}, diff --git a/configure b/configure --- a/configure +++ b/configure @@ -8551,6 +8551,48 @@ fi # Dynamic linking for HP-UX +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for RAND_egd in -lcrypto" >&5 +$as_echo_n "checking for RAND_egd in -lcrypto... " >&6; } +if ${ac_cv_lib_crypto_RAND_egd+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lcrypto $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char RAND_egd (); +int +main () +{ +return RAND_egd (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_crypto_RAND_egd=yes +else + ac_cv_lib_crypto_RAND_egd=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_crypto_RAND_egd" >&5 +$as_echo "$ac_cv_lib_crypto_RAND_egd" >&6; } +if test "x$ac_cv_lib_crypto_RAND_egd" = xyes; then : + +$as_echo "#define HAVE_RAND_EGD 1" >>confdefs.h + +fi + # only check for sem_init if thread support is requested if test "$with_threads" = "yes" -o -z "$with_threads"; then diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -2221,6 +2221,9 @@ # checks for libraries AC_CHECK_LIB(dl, dlopen) # Dynamic linking for SunOS/Solaris and SYSV AC_CHECK_LIB(dld, shl_load) # Dynamic linking for HP-UX +AC_CHECK_LIB(crypto, RAND_egd, + AC_DEFINE(HAVE_RAND_EGD, 1, + [Define if the libcrypto has RAND_egd])) # only check for sem_init if thread support is requested if test "$with_threads" = "yes" -o -z "$with_threads"; then diff --git a/pyconfig.h.in b/pyconfig.h.in --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -547,6 +547,9 @@ /* Define to 1 if you have the `putenv' function. */ #undef HAVE_PUTENV +/* Define if the libcrypto has RAND_egd */ +#undef HAVE_RAND_EGD + /* Define to 1 if you have the `readlink' function. */ #undef HAVE_READLINK -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 14:00:50 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 13:00:50 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMzU2?= =?utf-8?q?=3A_Make_ssl=2ERAND=5Fegd=28=29_optional_to_support_LibreSSL=2E?= =?utf-8?q?_The?= Message-ID: <20150106130030.72551.63511@psf.io> https://hg.python.org/cpython/rev/7f82f50fdad0 changeset: 94050:7f82f50fdad0 branch: 3.4 parent: 94045:a8c4925e2359 user: Victor Stinner date: Tue Jan 06 13:54:58 2015 +0100 summary: Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The availability of the function is checked during the compilation. Patch written by Bernard Spil. files: Lib/ssl.py | 7 ++++- Lib/test/test_ssl.py | 5 ++- Misc/NEWS | 4 +++ Modules/_ssl.c | 4 +++ configure | 42 ++++++++++++++++++++++++++++++++ configure.ac | 3 ++ pyconfig.h.in | 3 ++ 7 files changed, 65 insertions(+), 3 deletions(-) diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -106,7 +106,12 @@ from _ssl import (VERIFY_DEFAULT, VERIFY_CRL_CHECK_LEAF, VERIFY_CRL_CHECK_CHAIN, VERIFY_X509_STRICT) from _ssl import txt2obj as _txt2obj, nid2obj as _nid2obj -from _ssl import RAND_status, RAND_egd, RAND_add, RAND_bytes, RAND_pseudo_bytes +from _ssl import RAND_status, RAND_add, RAND_bytes, RAND_pseudo_bytes +try: + from _ssl import RAND_egd +except ImportError: + # LibreSSL does not provide RAND_egd + pass def _import_symbols(prefix): for n in dir(_ssl): diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -154,8 +154,9 @@ self.assertRaises(ValueError, ssl.RAND_bytes, -5) self.assertRaises(ValueError, ssl.RAND_pseudo_bytes, -5) - self.assertRaises(TypeError, ssl.RAND_egd, 1) - self.assertRaises(TypeError, ssl.RAND_egd, 'foo', 1) + if hasattr(ssl, 'RAND_egd'): + self.assertRaises(TypeError, ssl.RAND_egd, 1) + self.assertRaises(TypeError, ssl.RAND_egd, 'foo', 1) ssl.RAND_add("this is a random string", 75.0) @unittest.skipUnless(os.name == 'posix', 'requires posix') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,10 @@ Library ------- +- Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The + availability of the function is checked during the compilation. Patch written + by Bernard Spil. + - Issue #20896, #22935: The :func:`ssl.get_server_certificate` function now uses the :data:`~ssl.PROTOCOL_SSLv23` protocol by default, not :data:`~ssl.PROTOCOL_SSLv3`, for maximum compatibility and support platforms diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -3335,6 +3335,7 @@ It is necessary to seed the PRNG with RAND_add() on some platforms before\n\ using the ssl() function."); +#ifdef HAVE_RAND_EGD static PyObject * PySSL_RAND_egd(PyObject *self, PyObject *args) { @@ -3362,6 +3363,7 @@ Queries the entropy gather daemon (EGD) on the socket named by 'path'.\n\ Returns number of bytes read. Raises SSLError if connection to EGD\n\ fails or if it does not provide enough data to seed PRNG."); +#endif /* HAVE_RAND_EGD */ #endif /* HAVE_OPENSSL_RAND */ @@ -3757,8 +3759,10 @@ PySSL_RAND_bytes_doc}, {"RAND_pseudo_bytes", PySSL_RAND_pseudo_bytes, METH_VARARGS, PySSL_RAND_pseudo_bytes_doc}, +#ifdef HAVE_RAND_EGD {"RAND_egd", PySSL_RAND_egd, METH_VARARGS, PySSL_RAND_egd_doc}, +#endif {"RAND_status", (PyCFunction)PySSL_RAND_status, METH_NOARGS, PySSL_RAND_status_doc}, #endif diff --git a/configure b/configure --- a/configure +++ b/configure @@ -8913,6 +8913,48 @@ fi # Dynamic linking for HP-UX +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for RAND_egd in -lcrypto" >&5 +$as_echo_n "checking for RAND_egd in -lcrypto... " >&6; } +if ${ac_cv_lib_crypto_RAND_egd+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lcrypto $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char RAND_egd (); +int +main () +{ +return RAND_egd (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_crypto_RAND_egd=yes +else + ac_cv_lib_crypto_RAND_egd=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_crypto_RAND_egd" >&5 +$as_echo "$ac_cv_lib_crypto_RAND_egd" >&6; } +if test "x$ac_cv_lib_crypto_RAND_egd" = xyes; then : + +$as_echo "#define HAVE_RAND_EGD 1" >>confdefs.h + +fi + # only check for sem_init if thread support is requested if test "$with_threads" = "yes" -o -z "$with_threads"; then diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -2238,6 +2238,9 @@ AC_CHECK_LIB(sendfile, sendfile) AC_CHECK_LIB(dl, dlopen) # Dynamic linking for SunOS/Solaris and SYSV AC_CHECK_LIB(dld, shl_load) # Dynamic linking for HP-UX +AC_CHECK_LIB(crypto, RAND_egd, + AC_DEFINE(HAVE_RAND_EGD, 1, + [Define if the libcrypto has RAND_egd])) # only check for sem_init if thread support is requested if test "$with_threads" = "yes" -o -z "$with_threads"; then diff --git a/pyconfig.h.in b/pyconfig.h.in --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -675,6 +675,9 @@ /* Define to 1 if you have the `pwrite' function. */ #undef HAVE_PWRITE +/* Define if the libcrypto has RAND_egd */ +#undef HAVE_RAND_EGD + /* Define to 1 if you have the `readlink' function. */ #undef HAVE_READLINK -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 14:10:24 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 06 Jan 2015 13:10:24 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323145=3A_regrtest?= =?utf-8?q?_now_shows_errors_and_raises_an_exception_if?= Message-ID: <20150106130922.125906.46392@psf.io> https://hg.python.org/cpython/rev/1bb3df5bb83f changeset: 94052:1bb3df5bb83f user: Victor Stinner date: Tue Jan 06 14:05:03 2015 +0100 summary: Issue #23145: regrtest now shows errors and raises an exception if loader.loadTestsFromModule() logged errors. files: Lib/test/regrtest.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -1276,6 +1276,10 @@ def test_runner(): loader = unittest.TestLoader() tests = loader.loadTestsFromModule(the_module) + for error in loader.errors: + print(error, file=sys.stderr) + if loader.errors: + raise Exception("errors while loading tests") support.run_unittest(tests) test_runner() if huntrleaks: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 15:38:52 2015 From: python-checkins at python.org (nick.coghlan) Date: Tue, 06 Jan 2015 14:38:52 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_issue_19548_changes_from_3=2E4?= Message-ID: <20150106143717.125886.66642@psf.io> https://hg.python.org/cpython/rev/4d00d0109147 changeset: 94054:4d00d0109147 parent: 94052:1bb3df5bb83f parent: 94053:0646eee8296a user: Nick Coghlan date: Wed Jan 07 00:37:01 2015 +1000 summary: Merge issue 19548 changes from 3.4 files: Doc/glossary.rst | 5 +- Doc/library/codecs.rst | 667 +++++++++++++------------ Doc/library/functions.rst | 8 +- Doc/library/stdtypes.rst | 4 +- Doc/library/tarfile.rst | 2 +- Lib/codecs.py | 71 +- Lib/test/test_codecs.py | 46 +- Misc/NEWS | 8 + Modules/_codecsmodule.c | 6 +- 9 files changed, 432 insertions(+), 385 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -834,10 +834,13 @@ :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences include :data:`sys.float_info` and the return value of :func:`os.stat`. + text encoding + A codec which encodes Unicode strings to bytes. + text file A :term:`file object` able to read and write :class:`str` objects. Often, a text file actually accesses a byte-oriented datastream - and handles the text encoding automatically. + and handles the :term:`text encoding` automatically. .. seealso:: A :term:`binary file` reads and write :class:`bytes` objects. diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -18,17 +18,24 @@ pair: stackable; streams This module defines base classes for standard Python codecs (encoders and -decoders) and provides access to the internal Python codec registry which -manages the codec and error handling lookup process. +decoders) and provides access to the internal Python codec registry, which +manages the codec and error handling lookup process. Most standard codecs +are :term:`text encodings `, which encode text to bytes, +but there are also codecs provided that encode text to text, and bytes to +bytes. Custom codecs may encode and decode between arbitrary types, but some +module features are restricted to use specifically with +:term:`text encodings `, or with codecs that encode to +:class:`bytes`. -It defines the following functions: +The module defines the following functions for encoding and decoding with +any codec: .. function:: encode(obj, encoding='utf-8', errors='strict') Encodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The - default error handler is ``strict`` meaning that encoding errors raise + default error handler is ``'strict'`` meaning that encoding errors raise :exc:`ValueError` (or a more codec specific subclass, such as :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. @@ -38,93 +45,63 @@ Decodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The - default error handler is ``strict`` meaning that decoding errors raise + default error handler is ``'strict'`` meaning that decoding errors raise :exc:`ValueError` (or a more codec specific subclass, such as :exc:`UnicodeDecodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. -.. function:: register(search_function) - - Register a codec search function. Search functions are expected to take one - argument, the encoding name in all lower case letters, and return a - :class:`CodecInfo` object having the following attributes: - - * ``name`` The name of the encoding; - - * ``encode`` The stateless encoding function; - - * ``decode`` The stateless decoding function; - - * ``incrementalencoder`` An incremental encoder class or factory function; - - * ``incrementaldecoder`` An incremental decoder class or factory function; - - * ``streamwriter`` A stream writer class or factory function; - - * ``streamreader`` A stream reader class or factory function. - - The various functions or classes take the following arguments: - - *encode* and *decode*: These must be functions or methods which have the same - interface as the :meth:`~Codec.encode`/:meth:`~Codec.decode` methods of Codec - instances (see :ref:`Codec Interface `). The functions/methods - are expected to work in a stateless mode. - - *incrementalencoder* and *incrementaldecoder*: These have to be factory - functions providing the following interface: - - ``factory(errors='strict')`` - - The factory functions must return objects providing the interfaces defined by - the base classes :class:`IncrementalEncoder` and :class:`IncrementalDecoder`, - respectively. Incremental codecs can maintain state. - - *streamreader* and *streamwriter*: These have to be factory functions providing - the following interface: - - ``factory(stream, errors='strict')`` - - The factory functions must return objects providing the interfaces defined by - the base classes :class:`StreamReader` and :class:`StreamWriter`, respectively. - Stream codecs can maintain state. - - Possible values for errors are - - * ``'strict'``: raise an exception in case of an encoding error - * ``'replace'``: replace malformed data with a suitable replacement marker, - such as ``'?'`` or ``'\ufffd'`` - * ``'ignore'``: ignore malformed data and continue without further notice - * ``'xmlcharrefreplace'``: replace with the appropriate XML character - reference (for encoding only) - * ``'backslashreplace'``: replace with backslashed escape sequences (for - encoding only) - * ``'namereplace'``: replace with ``\N{...}`` escape sequences (for - encoding only) - * ``'surrogateescape'``: on decoding, replace with code points in the Unicode - Private Use Area ranging from U+DC80 to U+DCFF. These private code - points will then be turned back into the same bytes when the - ``surrogateescape`` error handler is used when encoding the data. - (See :pep:`383` for more.) - - as well as any other error handling name defined via :func:`register_error`. - - In case a search function cannot find a given encoding, it should return - ``None``. - +The full details for each codec can also be looked up directly: .. function:: lookup(encoding) Looks up the codec info in the Python codec registry and returns a - :class:`CodecInfo` object as defined above. + :class:`CodecInfo` object as defined below. Encodings are first looked up in the registry's cache. If not found, the list of registered search functions is scanned. If no :class:`CodecInfo` object is found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object is stored in the cache and returned to the caller. -To simplify access to the various codecs, the module provides these additional -functions which use :func:`lookup` for the codec lookup: +.. class:: CodecInfo(encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None) + Codec details when looking up the codec registry. The constructor + arguments are stored in attributes of the same name: + + + .. attribute:: name + + The name of the encoding. + + + .. attribute:: encode + decode + + The stateless encoding and decoding functions. These must be + functions or methods which have the same interface as + the :meth:`~Codec.encode` and :meth:`~Codec.decode` methods of Codec + instances (see :ref:`Codec Interface `). + The functions or methods are expected to work in a stateless mode. + + + .. attribute:: incrementalencoder + incrementaldecoder + + Incremental encoder and decoder classes or factory functions. + These have to provide the interface defined by the base classes + :class:`IncrementalEncoder` and :class:`IncrementalDecoder`, + respectively. Incremental codecs can maintain state. + + + .. attribute:: streamwriter + streamreader + + Stream writer and reader classes or factory functions. These have to + provide the interface defined by the base classes + :class:`StreamWriter` and :class:`StreamReader`, respectively. + Stream codecs can maintain state. + +To simplify access to the various codec components, the module provides +these additional functions which use :func:`lookup` for the codec lookup: .. function:: getencoder(encoding) @@ -173,97 +150,43 @@ Raises a :exc:`LookupError` in case the encoding cannot be found. +Custom codecs are made available by registering a suitable codec search +function: -.. function:: register_error(name, error_handler) +.. function:: register(search_function) - Register the error handling function *error_handler* under the name *name*. - *error_handler* will be called during encoding and decoding in case of an error, - when *name* is specified as the errors parameter. - - For encoding *error_handler* will be called with a :exc:`UnicodeEncodeError` - instance, which contains information about the location of the error. The - error handler must either raise this or a different exception or return a - tuple with a replacement for the unencodable part of the input and a position - where encoding should continue. The replacement may be either :class:`str` or - :class:`bytes`. If the replacement is bytes, the encoder will simply copy - them into the output buffer. If the replacement is a string, the encoder will - encode the replacement. Encoding continues on original input at the - specified position. Negative position values will be treated as being - relative to the end of the input string. If the resulting position is out of - bound an :exc:`IndexError` will be raised. - - Decoding and translating works similar, except :exc:`UnicodeDecodeError` or - :exc:`UnicodeTranslateError` will be passed to the handler and that the - replacement from the error handler will be put into the output directly. - - -.. function:: lookup_error(name) - - Return the error handler previously registered under the name *name*. - - Raises a :exc:`LookupError` in case the handler cannot be found. - - -.. function:: strict_errors(exception) - - Implements the ``strict`` error handling: each encoding or decoding error - raises a :exc:`UnicodeError`. - - -.. function:: replace_errors(exception) - - Implements the ``replace`` error handling: malformed data is replaced with a - suitable replacement character such as ``'?'`` in bytestrings and - ``'\ufffd'`` in Unicode strings. - - -.. function:: ignore_errors(exception) - - Implements the ``ignore`` error handling: malformed data is ignored and - encoding or decoding is continued without further notice. - - -.. function:: xmlcharrefreplace_errors(exception) - - Implements the ``xmlcharrefreplace`` error handling (for encoding only): the - unencodable character is replaced by an appropriate XML character reference. - - -.. function:: backslashreplace_errors(exception) - - Implements the ``backslashreplace`` error handling (for encoding only): the - unencodable character is replaced by a backslashed escape sequence. - -.. function:: namereplace_errors(exception) - - Implements the ``namereplace`` error handling (for encoding only): the - unencodable character is replaced by a ``\N{...}`` escape sequence. - - .. versionadded:: 3.5 - -To simplify working with encoded files or stream, the module also defines these -utility functions: - - -.. function:: open(filename, mode[, encoding[, errors[, buffering]]]) - - Open an encoded file using the given *mode* and return a wrapped version - providing transparent encoding/decoding. The default file mode is ``'r'`` - meaning to open the file in read mode. + Register a codec search function. Search functions are expected to take one + argument, being the encoding name in all lower case letters, and return a + :class:`CodecInfo` object. In case a search function cannot find + a given encoding, it should return ``None``. .. note:: - The wrapped version's methods will accept and return strings only. Bytes - arguments will be rejected. + Search function registration is not currently reversible, + which may cause problems in some cases, such as unit testing or + module reloading. + +While the builtin :func:`open` and the associated :mod:`io` module are the +recommended approach for working with encoded text files, this module +provides additional utility functions and classes that allow the use of a +wider range of codecs when working with binary files: + +.. function:: open(filename, mode='r', encoding=None, errors='strict', buffering=1) + + Open an encoded file using the given *mode* and return an instance of + :class:`StreamReaderWriter`, providing transparent encoding/decoding. + The default file mode is ``'r'``, meaning to open the file in read mode. .. note:: - Files are always opened in binary mode, even if no binary mode was - specified. This is done to avoid data loss due to encodings using 8-bit - values. This means that no automatic conversion of ``b'\n'`` is done - on reading and writing. + Underlying encoded files are always opened in binary mode. + No automatic conversion of ``'\n'`` is done on reading and writing. + The *mode* argument may be any binary mode acceptable to the built-in + :func:`open` function; the ``'b'`` is automatically added. *encoding* specifies the encoding which is to be used for the file. + Any encoding that encodes to and decodes from bytes is allowed, and + the data types supported by the file methods depend on the codec used. *errors* may be given to define the error handling. It defaults to ``'strict'`` which causes a :exc:`ValueError` to be raised in case an encoding error occurs. @@ -274,12 +197,15 @@ .. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict') - Return a wrapped version of file which provides transparent encoding - translation. + Return a :class:`StreamRecoder` instance, a wrapped version of *file* + which provides transparent transcoding. The original file is closed + when the wrapped version is closed. - Bytes written to the wrapped file are interpreted according to the given - *data_encoding* and then written to the original file as bytes using the - *file_encoding*. + Data written to the wrapped file is decoded according to the given + *data_encoding* and then written to the original file as bytes using + *file_encoding*. Bytes read from the original file are decoded + according to *file_encoding*, and the result is encoded + using *data_encoding*. If *file_encoding* is not given, it defaults to *data_encoding*. @@ -291,14 +217,16 @@ .. function:: iterencode(iterator, encoding, errors='strict', **kwargs) Uses an incremental encoder to iteratively encode the input provided by - *iterator*. This function is a :term:`generator`. *errors* (as well as any + *iterator*. This function is a :term:`generator`. + The *errors* argument (as well as any other keyword argument) is passed through to the incremental encoder. .. function:: iterdecode(iterator, encoding, errors='strict', **kwargs) Uses an incremental decoder to iteratively decode the input provided by - *iterator*. This function is a :term:`generator`. *errors* (as well as any + *iterator*. This function is a :term:`generator`. + The *errors* argument (as well as any other keyword argument) is passed through to the incremental decoder. @@ -317,9 +245,10 @@ BOM_UTF32_BE BOM_UTF32_LE - These constants define various encodings of the Unicode byte order mark (BOM) - used in UTF-16 and UTF-32 data streams to indicate the byte order used in the - stream or file and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either + These constants define various byte sequences, + being Unicode byte order marks (BOMs) for several encodings. They are + used in UTF-16 and UTF-32 data streams to indicate the byte order used, + and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`, :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for @@ -334,20 +263,25 @@ ------------------ The :mod:`codecs` module defines a set of base classes which define the -interface and can also be used to easily write your own codecs for use in -Python. +interfaces for working with codec objects, and can also be used as the basis +for custom codec implementations. Each codec has to define four interfaces to make it usable as codec in Python: stateless encoder, stateless decoder, stream reader and stream writer. The stream reader and writers typically reuse the stateless encoder/decoder to -implement the file protocols. +implement the file protocols. Codec authors also need to define how the +codec will handle encoding and decoding errors. -The :class:`Codec` class defines the interface for stateless encoders/decoders. -To simplify and standardize error handling, the :meth:`~Codec.encode` and -:meth:`~Codec.decode` methods may implement different error handling schemes by -providing the *errors* string argument. The following string values are defined -and implemented by all standard Python codecs: +.. _error-handlers: + +Error Handlers +^^^^^^^^^^^^^^ + +To simplify and standardize error handling, +codecs may implement different error handling schemes by +accepting the *errors* string argument. The following string values are +defined and implemented by all standard Python codecs: .. tabularcolumns:: |l|L| @@ -355,39 +289,55 @@ | Value | Meaning | +=========================+===============================================+ | ``'strict'`` | Raise :exc:`UnicodeError` (or a subclass); | -| | this is the default. | +| | this is the default. Implemented in | +| | :func:`strict_errors`. | +-------------------------+-----------------------------------------------+ -| ``'ignore'`` | Ignore the character and continue with the | -| | next. | +| ``'ignore'`` | Ignore the malformed data and continue | +| | without further notice. Implemented in | +| | :func:`ignore_errors`. | +-------------------------+-----------------------------------------------+ + +The following error handlers are only applicable to +:term:`text encodings `: + ++-------------------------+-----------------------------------------------+ +| Value | Meaning | ++=========================+===============================================+ | ``'replace'`` | Replace with a suitable replacement | -| | character; Python will use the official | -| | U+FFFD REPLACEMENT CHARACTER for the built-in | -| | Unicode codecs on decoding and '?' on | -| | encoding. | +| | marker; Python will use the official | +| | ``U+FFFD`` REPLACEMENT CHARACTER for the | +| | built-in codecs on decoding, and '?' on | +| | encoding. Implemented in | +| | :func:`replace_errors`. | +-------------------------+-----------------------------------------------+ | ``'xmlcharrefreplace'`` | Replace with the appropriate XML character | -| | reference (only for encoding). | +| | reference (only for encoding). Implemented | +| | in :func:`xmlcharrefreplace_errors`. | +-------------------------+-----------------------------------------------+ | ``'backslashreplace'`` | Replace with backslashed escape sequences | -| | (only for encoding). | +| | (only for encoding). Implemented in | +| | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ | ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | | | (only for encoding). | +-------------------------+-----------------------------------------------+ -| ``'surrogateescape'`` | Replace byte with surrogate U+DCxx, as defined| -| | in :pep:`383`. | +| ``'surrogateescape'`` | On decoding, replace byte with individual | +| | surrogate code ranging from ``U+DC80`` to | +| | ``U+DCFF``. This code will then be turned | +| | back into the same byte when the | +| | ``'surrogateescape'`` error handler is used | +| | when encoding the data. (See :pep:`383` for | +| | more.) | +-------------------------+-----------------------------------------------+ -In addition, the following error handlers are specific to Unicode encoding -schemes: +In addition, the following error handler is specific to the given codecs: +-------------------+------------------------+-------------------------------------------+ -| Value | Codec | Meaning | +| Value | Codecs | Meaning | +===================+========================+===========================================+ |``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding of surrogate | -| | utf-16-be, utf-16-le, | codes in all the Unicode encoding schemes.| -| | utf-32-be, utf-32-le | | +| | utf-16-be, utf-16-le, | codes. These codecs normally treat the | +| | utf-32-be, utf-32-le | presence of surrogates as an error. | +-------------------+------------------------+-------------------------------------------+ .. versionadded:: 3.1 @@ -399,26 +349,103 @@ .. versionadded:: 3.5 The ``'namereplace'`` error handler. -The set of allowed values can be extended via :meth:`register_error`. +The set of allowed values can be extended by registering a new named error +handler: + +.. function:: register_error(name, error_handler) + + Register the error handling function *error_handler* under the name *name*. + The *error_handler* argument will be called during encoding and decoding + in case of an error, when *name* is specified as the errors parameter. + + For encoding, *error_handler* will be called with a :exc:`UnicodeEncodeError` + instance, which contains information about the location of the error. The + error handler must either raise this or a different exception, or return a + tuple with a replacement for the unencodable part of the input and a position + where encoding should continue. The replacement may be either :class:`str` or + :class:`bytes`. If the replacement is bytes, the encoder will simply copy + them into the output buffer. If the replacement is a string, the encoder will + encode the replacement. Encoding continues on original input at the + specified position. Negative position values will be treated as being + relative to the end of the input string. If the resulting position is out of + bound an :exc:`IndexError` will be raised. + + Decoding and translating works similarly, except :exc:`UnicodeDecodeError` or + :exc:`UnicodeTranslateError` will be passed to the handler and that the + replacement from the error handler will be put into the output directly. + + +Previously registered error handlers (including the standard error handlers) +can be looked up by name: + +.. function:: lookup_error(name) + + Return the error handler previously registered under the name *name*. + + Raises a :exc:`LookupError` in case the handler cannot be found. + +The following standard error handlers are also made available as module level +functions: + +.. function:: strict_errors(exception) + + Implements the ``'strict'`` error handling: each encoding or + decoding error raises a :exc:`UnicodeError`. + + +.. function:: replace_errors(exception) + + Implements the ``'replace'`` error handling (for :term:`text encodings + ` only): substitutes ``'?'`` for encoding errors + (to be encoded by the codec), and ``'\ufffd'`` (the Unicode replacement + character, ``'?'``) for decoding errors. + + +.. function:: ignore_errors(exception) + + Implements the ``'ignore'`` error handling: malformed data is ignored and + encoding or decoding is continued without further notice. + + +.. function:: xmlcharrefreplace_errors(exception) + + Implements the ``'xmlcharrefreplace'`` error handling (for encoding with + :term:`text encodings ` only): the + unencodable character is replaced by an appropriate XML character reference. + + +.. function:: backslashreplace_errors(exception) + + Implements the ``'backslashreplace'`` error handling (for encoding with + :term:`text encodings ` only): the + unencodable character is replaced by a backslashed escape sequence. + +.. function:: namereplace_errors(exception) + + Implements the ``namereplace`` error handling (for encoding only): the + unencodable character is replaced by a ``\N{...}`` escape sequence. + + .. versionadded:: 3.5 .. _codec-objects: -Codec Objects -^^^^^^^^^^^^^ +Stateless Encoding and Decoding +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The :class:`Codec` class defines these methods which also define the function -interfaces of the stateless encoder and decoder: +The base :class:`Codec` class defines these methods which also define the +function interfaces of the stateless encoder and decoder: .. method:: Codec.encode(input[, errors]) Encodes the object *input* and returns a tuple (output object, length consumed). - Encoding converts a string object to a bytes object using a particular + For instance, :term:`text encoding` converts + a string object to a bytes object using a particular character set encoding (e.g., ``cp1252`` or ``iso-8859-1``). - *errors* defines the error handling to apply. It defaults to ``'strict'`` - handling. + The *errors* argument defines the error handling to apply. + It defaults to ``'strict'`` handling. The method may not store state in the :class:`Codec` instance. Use :class:`StreamCodec` for codecs which have to keep state in order to make @@ -431,14 +458,16 @@ .. method:: Codec.decode(input[, errors]) Decodes the object *input* and returns a tuple (output object, length - consumed). Decoding converts a bytes object encoded using a particular + consumed). For instance, for a :term:`text encoding`, decoding converts + a bytes object encoded using a particular character set encoding to a string object. - *input* must be a bytes object or one which provides the read-only character + For text encodings and bytes-to-bytes codecs, + *input* must be a bytes object or one which provides the read-only buffer interface -- for example, buffer objects and memory mapped files. - *errors* defines the error handling to apply. It defaults to ``'strict'`` - handling. + The *errors* argument defines the error handling to apply. + It defaults to ``'strict'`` handling. The method may not store state in the :class:`Codec` instance. Use :class:`StreamCodec` for codecs which have to keep state in order to make @@ -447,6 +476,10 @@ The decoder must be able to handle zero length input and return an empty object of the output object type in this situation. + +Incremental Encoding and Decoding +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide the basic interface for incremental encoding and decoding. Encoding/decoding the input isn't done with one call to the stateless encoder/decoder function, but @@ -464,14 +497,14 @@ .. _incremental-encoder-objects: IncrementalEncoder Objects -^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~ The :class:`IncrementalEncoder` class is used for encoding an input in multiple steps. It defines the following methods which every incremental encoder must define in order to be compatible with the Python codec registry. -.. class:: IncrementalEncoder([errors]) +.. class:: IncrementalEncoder(errors='strict') Constructor for an :class:`IncrementalEncoder` instance. @@ -480,28 +513,14 @@ the Python codec registry. The :class:`IncrementalEncoder` may implement different error handling schemes - by providing the *errors* keyword argument. These parameters are predefined: - - * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default. - - * ``'ignore'`` Ignore the character and continue with the next. - - * ``'replace'`` Replace with a suitable replacement character - - * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference - - * ``'backslashreplace'`` Replace with backslashed escape sequences. - - * ``'namereplace'`` Replace with ``\N{...}`` escape sequences. + by providing the *errors* keyword argument. See :ref:`error-handlers` for + possible values. The *errors* argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`IncrementalEncoder` object. - The set of allowed values for the *errors* argument can be extended with - :func:`register_error`. - .. method:: encode(object[, final]) @@ -513,7 +532,8 @@ .. method:: reset() Reset the encoder to the initial state. The output is discarded: call - ``.encode('', final=True)`` to reset the encoder and to get the output. + ``.encode(object, final=True)``, passing an empty byte or text string + if necessary, to reset the encoder and to get the output. .. method:: IncrementalEncoder.getstate() @@ -534,14 +554,14 @@ .. _incremental-decoder-objects: IncrementalDecoder Objects -^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~ The :class:`IncrementalDecoder` class is used for decoding an input in multiple steps. It defines the following methods which every incremental decoder must define in order to be compatible with the Python codec registry. -.. class:: IncrementalDecoder([errors]) +.. class:: IncrementalDecoder(errors='strict') Constructor for an :class:`IncrementalDecoder` instance. @@ -550,22 +570,14 @@ the Python codec registry. The :class:`IncrementalDecoder` may implement different error handling schemes - by providing the *errors* keyword argument. These parameters are predefined: - - * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default. - - * ``'ignore'`` Ignore the character and continue with the next. - - * ``'replace'`` Replace with a suitable replacement character. + by providing the *errors* keyword argument. See :ref:`error-handlers` for + possible values. The *errors* argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`IncrementalDecoder` object. - The set of allowed values for the *errors* argument can be extended with - :func:`register_error`. - .. method:: decode(object[, final]) @@ -604,6 +616,10 @@ returned by :meth:`getstate`. +Stream Encoding and Decoding +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + The :class:`StreamWriter` and :class:`StreamReader` classes provide generic working interfaces which can be used to implement new encoding submodules very easily. See :mod:`encodings.utf_8` for an example of how this is done. @@ -612,14 +628,14 @@ .. _stream-writer-objects: StreamWriter Objects -^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~ The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the following methods which every stream writer must define in order to be compatible with the Python codec registry. -.. class:: StreamWriter(stream[, errors]) +.. class:: StreamWriter(stream, errors='strict') Constructor for a :class:`StreamWriter` instance. @@ -627,31 +643,17 @@ additional keyword arguments, but only the ones defined here are used by the Python codec registry. - *stream* must be a file-like object open for writing binary data. + The *stream* argument must be a file-like object open for writing + text or binary data, as appropriate for the specific codec. The :class:`StreamWriter` may implement different error handling schemes by - providing the *errors* keyword argument. These parameters are predefined: - - * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default. - - * ``'ignore'`` Ignore the character and continue with the next. - - * ``'replace'`` Replace with a suitable replacement character - - * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference - - * ``'backslashreplace'`` Replace with backslashed escape sequences. - - * ``'namereplace'`` Replace with ``\N{...}`` escape sequences. + providing the *errors* keyword argument. See :ref:`error-handlers` for + the standard error handlers the underlying stream codec may support. The *errors* argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`StreamWriter` object. - The set of allowed values for the *errors* argument can be extended with - :func:`register_error`. - - .. method:: write(object) Writes the object's contents encoded to the stream. @@ -660,7 +662,8 @@ .. method:: writelines(list) Writes the concatenated list of strings to the stream (possibly by reusing - the :meth:`write` method). + the :meth:`write` method). The standard bytes-to-bytes codecs + do not support this method. .. method:: reset() @@ -679,14 +682,14 @@ .. _stream-reader-objects: StreamReader Objects -^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~ The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the following methods which every stream reader must define in order to be compatible with the Python codec registry. -.. class:: StreamReader(stream[, errors]) +.. class:: StreamReader(stream, errors='strict') Constructor for a :class:`StreamReader` instance. @@ -694,16 +697,12 @@ additional keyword arguments, but only the ones defined here are used by the Python codec registry. - *stream* must be a file-like object open for reading (binary) data. + The *stream* argument must be a file-like object open for reading + text or binary data, as appropriate for the specific codec. The :class:`StreamReader` may implement different error handling schemes by - providing the *errors* keyword argument. These parameters are defined: - - * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default. - - * ``'ignore'`` Ignore the character and continue with the next. - - * ``'replace'`` Replace with a suitable replacement character. + providing the *errors* keyword argument. See :ref:`error-handlers` for + the standard error handlers the underlying stream codec may support. The *errors* argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error @@ -717,17 +716,20 @@ Decodes data from the stream and returns the resulting object. - *chars* indicates the number of characters to read from the - stream. :func:`read` will never return more than *chars* characters, but - it might return less, if there are not enough characters available. + The *chars* argument indicates the number of decoded + code points or bytes to return. The :func:`read` method will + never return more data than requested, but it might return less, + if there is not enough available. - *size* indicates the approximate maximum number of bytes to read from the - stream for decoding purposes. The decoder can modify this setting as + The *size* argument indicates the approximate maximum + number of encoded bytes or code points to read + for decoding. The decoder can modify this setting as appropriate. The default value -1 indicates to read and decode as much as - possible. *size* is intended to prevent having to decode huge files in - one step. + possible. This parameter is intended to + prevent having to decode huge files in one step. - *firstline* indicates that it would be sufficient to only return the first + The *firstline* flag indicates that + it would be sufficient to only return the first line, if there are decoding errors on later lines. The method should use a greedy read strategy meaning that it should read @@ -770,17 +772,13 @@ In addition to the above methods, the :class:`StreamReader` must also inherit all other methods and attributes from the underlying stream. -The next two base classes are included for convenience. They are not needed by -the codec registry, but may provide useful in practice. - - .. _stream-reader-writer: StreamReaderWriter Objects -^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~ -The :class:`StreamReaderWriter` allows wrapping streams which work in both read -and write modes. +The :class:`StreamReaderWriter` is a convenience class that allows wrapping +streams which work in both read and write modes. The design is such that one can use the factory functions returned by the :func:`lookup` function to construct the instance. @@ -801,9 +799,9 @@ .. _stream-recoder-objects: StreamRecoder Objects -^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~ -The :class:`StreamRecoder` provide a frontend - backend view of encoding data +The :class:`StreamRecoder` translates data from one encoding to another, which is sometimes useful when dealing with different encoding environments. The design is such that one can use the factory functions returned by the @@ -813,22 +811,20 @@ .. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors) Creates a :class:`StreamRecoder` instance which implements a two-way conversion: - *encode* and *decode* work on the frontend (the input to :meth:`read` and output - of :meth:`write`) while *Reader* and *Writer* work on the backend (reading and - writing to the stream). + *encode* and *decode* work on the frontend?? the data visible to + code calling :meth:`read` and :meth:`write`, while *Reader* and *Writer* + work on the backend?? the data in *stream*. - You can use these objects to do transparent direct recodings from e.g. Latin-1 + You can use these objects to do transparent transcodings from e.g. Latin-1 to UTF-8 and back. - *stream* must be a file-like object. + The *stream* argument must be a file-like object. - *encode*, *decode* must adhere to the :class:`Codec` interface. *Reader*, + The *encode* and *decode* arguments must + adhere to the :class:`Codec` interface. *Reader* and *Writer* must be factory functions or classes providing objects of the :class:`StreamReader` and :class:`StreamWriter` interface respectively. - *encode* and *decode* are needed for the frontend translation, *Reader* and - *Writer* for the backend translation. - Error handling is done in the same way as defined for the stream readers and writers. @@ -843,20 +839,23 @@ Encodings and Unicode --------------------- -Strings are stored internally as sequences of codepoints in range ``0 - 10FFFF`` -(see :pep:`393` for more details about the implementation). -Once a string object is used outside of CPU and memory, CPU endianness -and how these arrays are stored as bytes become an issue. Transforming a -string object into a sequence of bytes is called encoding and recreating the -string object from the sequence of bytes is known as decoding. There are many -different methods for how this transformation can be done (these methods are -also called encodings). The simplest method is to map the codepoints 0-255 to -the bytes ``0x0``-``0xff``. This means that a string object that contains -codepoints above ``U+00FF`` can't be encoded with this method (which is called -``'latin-1'`` or ``'iso-8859-1'``). :func:`str.encode` will raise a -:exc:`UnicodeEncodeError` that looks like this: ``UnicodeEncodeError: 'latin-1' -codec can't encode character '\u1234' in position 3: ordinal not in -range(256)``. +Strings are stored internally as sequences of codepoints in +range ``0x0``-``0x10FFFF``. (See :pep:`393` for +more details about the implementation.) +Once a string object is used outside of CPU and memory, endianness +and how these arrays are stored as bytes become an issue. As with other +codecs, serialising a string into a sequence of bytes is known as *encoding*, +and recreating the string from the sequence of bytes is known as *decoding*. +There are a variety of different text serialisation codecs, which are +collectivity referred to as :term:`text encodings `. + +The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps +the codepoints 0-255 to the bytes ``0x0``-``0xff``, which means that a string +object that contains codepoints above ``U+00FF`` can't be encoded with this +codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks +like the following (although the details of the error message may differ): +``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in +position 3: ordinal not in range(256)``. There's another group of encodings (the so called charmap encodings) that choose a different subset of all Unicode code points and how these codepoints are @@ -1203,7 +1202,8 @@ .. versionchanged:: 3.4 The utf-16\* and utf-32\* encoders no longer allow surrogate code points - (U+D800--U+DFFF) to be encoded. The utf-32\* decoders no longer decode + (``U+D800``--``U+DFFF``) to be encoded. + The utf-32\* decoders no longer decode byte sequences that correspond to surrogate code points. @@ -1231,7 +1231,9 @@ +====================+=========+===========================+ | idna | | Implements :rfc:`3490`, | | | | see also | -| | | :mod:`encodings.idna` | +| | | :mod:`encodings.idna`. | +| | | Only ``errors='strict'`` | +| | | is supported. | +--------------------+---------+---------------------------+ | mbcs | dbcs | Windows only: Encode | | | | operand according to the | @@ -1239,31 +1241,44 @@ +--------------------+---------+---------------------------+ | palmos | | Encoding of PalmOS 3.5 | +--------------------+---------+---------------------------+ -| punycode | | Implements :rfc:`3492` | +| punycode | | Implements :rfc:`3492`. | +| | | Stateful codecs are not | +| | | supported. | +--------------------+---------+---------------------------+ -| raw_unicode_escape | | Produce a string that is | -| | | suitable as raw Unicode | -| | | literal in Python source | -| | | code | +| raw_unicode_escape | | Latin-1 encoding with | +| | | ``\uXXXX`` and | +| | | ``\UXXXXXXXX`` for other | +| | | code points. Existing | +| | | backslashes are not | +| | | escaped in any way. | +| | | It is used in the Python | +| | | pickle protocol. | +--------------------+---------+---------------------------+ | undefined | | Raise an exception for | -| | | all conversions. Can be | -| | | used as the system | -| | | encoding if no automatic | -| | | coercion between byte and | -| | | Unicode strings is | -| | | desired. | +| | | all conversions, even | +| | | empty strings. The error | +| | | handler is ignored. | +--------------------+---------+---------------------------+ -| unicode_escape | | Produce a string that is | -| | | suitable as Unicode | -| | | literal in Python source | -| | | code | +| unicode_escape | | Encoding suitable as the | +| | | contents of a Unicode | +| | | literal in ASCII-encoded | +| | | Python source code, | +| | | except that quotes are | +| | | not escaped. Decodes from | +| | | Latin-1 source code. | +| | | Beware that Python source | +| | | code actually uses UTF-8 | +| | | by default. | +--------------------+---------+---------------------------+ | unicode_internal | | Return the internal | | | | representation of the | -| | | operand | +| | | operand. Stateful codecs | +| | | are not supported. | | | | | | | | .. deprecated:: 3.3 | +| | | This representation is | +| | | obsoleted by | +| | | :pep:`393`. | +--------------------+---------+---------------------------+ .. _binary-transforms: @@ -1272,7 +1287,8 @@ ^^^^^^^^^^^^^^^^^ The following codecs provide binary transforms: :term:`bytes-like object` -to :class:`bytes` mappings. +to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode` +(which only produces :class:`str` output). .. tabularcolumns:: |l|L|L|L| @@ -1327,7 +1343,8 @@ ^^^^^^^^^^^^^^^ The following codec provides a text transform: a :class:`str` to :class:`str` -mapping. +mapping. It is not supported by :meth:`str.encode` (which only produces +:class:`bytes` output). .. tabularcolumns:: |l|l|L| diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -940,15 +940,17 @@ *encoding* is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever :func:`locale.getpreferredencoding` returns), but any - encoding supported by Python can be used. See the :mod:`codecs` module for + :term:`text encoding` supported by Python + can be used. See the :mod:`codecs` module for the list of supported encodings. *errors* is an optional string that specifies how encoding and decoding errors are to be handled--this cannot be used in binary mode. - A variety of standard error handlers are available, though any + A variety of standard error handlers are available + (listed under :ref:`error-handlers`), though any error handling name that has been registered with :func:`codecs.register_error` is also valid. The standard names - are: + include: * ``'strict'`` to raise a :exc:`ValueError` exception if there is an encoding error. The default value of ``None`` has the same diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1512,7 +1512,7 @@ a :exc:`UnicodeError`. Other possible values are ``'ignore'``, ``'replace'``, ``'xmlcharrefreplace'``, ``'backslashreplace'`` and any other name registered via - :func:`codecs.register_error`, see section :ref:`codec-base-classes`. For a + :func:`codecs.register_error`, see section :ref:`error-handlers`. For a list of possible encodings, see section :ref:`standard-encodings`. .. versionchanged:: 3.1 @@ -2384,7 +2384,7 @@ error handling scheme. The default for *errors* is ``'strict'``, meaning that encoding errors raise a :exc:`UnicodeError`. Other possible values are ``'ignore'``, ``'replace'`` and any other name registered via - :func:`codecs.register_error`, see section :ref:`codec-base-classes`. For a + :func:`codecs.register_error`, see section :ref:`error-handlers`. For a list of possible encodings, see section :ref:`standard-encodings`. .. note:: diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -798,7 +798,7 @@ appropriately, this conversion may fail. The *errors* argument defines how characters are treated that cannot be -converted. Possible values are listed in section :ref:`codec-base-classes`. +converted. Possible values are listed in section :ref:`error-handlers`. The default scheme is ``'surrogateescape'`` which Python also uses for its file system calls, see :ref:`os-filenames`. diff --git a/Lib/codecs.py b/Lib/codecs.py --- a/Lib/codecs.py +++ b/Lib/codecs.py @@ -347,8 +347,7 @@ """ Creates a StreamWriter instance. - stream must be a file-like object open for writing - (binary) data. + stream must be a file-like object open for writing. The StreamWriter may use different error handling schemes by providing the errors keyword argument. These @@ -422,8 +421,7 @@ """ Creates a StreamReader instance. - stream must be a file-like object open for reading - (binary) data. + stream must be a file-like object open for reading. The StreamReader may use different error handling schemes by providing the errors keyword argument. These @@ -451,13 +449,12 @@ """ Decodes data from the stream self.stream and returns the resulting object. - chars indicates the number of characters to read from the - stream. read() will never return more than chars - characters, but it might return less, if there are not enough - characters available. + chars indicates the number of decoded code points or bytes to + return. read() will never return more data than requested, + but it might return less, if there is not enough available. - size indicates the approximate maximum number of bytes to - read from the stream for decoding purposes. The decoder + size indicates the approximate maximum number of decoded + bytes or code points to read for decoding. The decoder can modify this setting as appropriate. The default value -1 indicates to read and decode as much as possible. size is intended to prevent having to decode huge files in one @@ -468,7 +465,7 @@ will be returned, the rest of the input will be kept until the next call to read(). - The method should use a greedy read strategy meaning that + The method should use a greedy read strategy, meaning that it should read as much data as is allowed within the definition of the encoding and the given size, e.g. if optional encoding endings or state markers are available @@ -603,7 +600,7 @@ def readlines(self, sizehint=None, keepends=True): """ Read all lines available on the input stream - and return them as list of lines. + and return them as a list. Line breaks are implemented using the codec's decoder method and are included in the list entries. @@ -751,19 +748,18 @@ class StreamRecoder: - """ StreamRecoder instances provide a frontend - backend - view of encoding data. + """ StreamRecoder instances translate data from one encoding to another. They use the complete set of APIs returned by the codecs.lookup() function to implement their task. - Data written to the stream is first decoded into an - intermediate format (which is dependent on the given codec - combination) and then written to the stream using an instance - of the provided Writer class. + Data written to the StreamRecoder is first decoded into an + intermediate format (depending on the "decode" codec) and then + written to the underlying stream using an instance of the provided + Writer class. - In the other direction, data is read from the stream using a - Reader instance and then return encoded data to the caller. + In the other direction, data is read from the underlying stream using + a Reader instance and then encoded and returned to the caller. """ # Optional attributes set by the file wrappers below @@ -775,22 +771,17 @@ """ Creates a StreamRecoder instance which implements a two-way conversion: encode and decode work on the frontend (the - input to .read() and output of .write()) while - Reader and Writer work on the backend (reading and - writing to the stream). + data visible to .read() and .write()) while Reader and Writer + work on the backend (the data in stream). - You can use these objects to do transparent direct - recodings from e.g. latin-1 to utf-8 and back. + You can use these objects to do transparent + transcodings from e.g. latin-1 to utf-8 and back. stream must be a file-like object. - encode, decode must adhere to the Codec interface, Reader, + encode and decode must adhere to the Codec interface; Reader and Writer must be factory functions or classes providing the - StreamReader, StreamWriter interface resp. - - encode and decode are needed for the frontend translation, - Reader and Writer for the backend translation. Unicode is - used as intermediate encoding. + StreamReader and StreamWriter interfaces resp. Error handling is done in the same way as defined for the StreamWriter/Readers. @@ -865,7 +856,7 @@ ### Shortcuts -def open(filename, mode='rb', encoding=None, errors='strict', buffering=1): +def open(filename, mode='r', encoding=None, errors='strict', buffering=1): """ Open an encoded file using the given mode and return a wrapped version providing transparent encoding/decoding. @@ -875,10 +866,8 @@ codecs. Output is also codec dependent and will usually be Unicode as well. - Files are always opened in binary mode, even if no binary mode - was specified. This is done to avoid data loss due to encodings - using 8-bit values. The default file mode is 'rb' meaning to - open the file in binary read mode. + Underlying encoded files are always opened in binary mode. + The default file mode is 'r', meaning to open the file in read mode. encoding specifies the encoding which is to be used for the file. @@ -914,13 +903,13 @@ """ Return a wrapped version of file which provides transparent encoding translation. - Strings written to the wrapped file are interpreted according - to the given data_encoding and then written to the original - file as string using file_encoding. The intermediate encoding + Data written to the wrapped file is decoded according + to the given data_encoding and then encoded to the underlying + file using file_encoding. The intermediate data type will usually be Unicode but depends on the specified codecs. - Strings are read from the file using file_encoding and then - passed back to the caller as string using data_encoding. + Bytes read from the file are decoded using file_encoding and then + passed back to the caller encoded using data_encoding. If file_encoding is not given, it defaults to data_encoding. diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1140,6 +1140,8 @@ # Python used to crash on this at exit because of a refcount # bug in _codecsmodule.c + self.assertTrue(f.closed) + # From RFC 3492 punycode_testcases = [ # A Arabic (Egyptian): @@ -1592,6 +1594,16 @@ self.assertEqual(encoder.encode("ample.org."), b"xn--xample-9ta.org.") self.assertEqual(encoder.encode("", True), b"") + def test_errors(self): + """Only supports "strict" error handler""" + "python.org".encode("idna", "strict") + b"python.org".decode("idna", "strict") + for errors in ("ignore", "replace", "backslashreplace", + "surrogateescape"): + self.assertRaises(Exception, "python.org".encode, "idna", errors) + self.assertRaises(Exception, + b"python.org".decode, "idna", errors) + class CodecsModuleTest(unittest.TestCase): def test_decode(self): @@ -1682,6 +1694,24 @@ for api in codecs.__all__: getattr(codecs, api) + def test_open(self): + self.addCleanup(support.unlink, support.TESTFN) + for mode in ('w', 'r', 'r+', 'w+', 'a', 'a+'): + with self.subTest(mode), \ + codecs.open(support.TESTFN, mode, 'ascii') as file: + self.assertIsInstance(file, codecs.StreamReaderWriter) + + def test_undefined(self): + self.assertRaises(UnicodeError, codecs.encode, 'abc', 'undefined') + self.assertRaises(UnicodeError, codecs.decode, b'abc', 'undefined') + self.assertRaises(UnicodeError, codecs.encode, '', 'undefined') + self.assertRaises(UnicodeError, codecs.decode, b'', 'undefined') + for errors in ('strict', 'ignore', 'replace', 'backslashreplace'): + self.assertRaises(UnicodeError, + codecs.encode, 'abc', 'undefined', errors) + self.assertRaises(UnicodeError, + codecs.decode, b'abc', 'undefined', errors) + class StreamReaderTest(unittest.TestCase): def setUp(self): @@ -1815,13 +1845,10 @@ # "undefined" # The following encodings don't work in stateful mode -broken_unicode_with_streams = [ +broken_unicode_with_stateful = [ "punycode", "unicode_internal" ] -broken_incremental_coders = broken_unicode_with_streams + [ - "idna", -] class BasicUnicodeTest(unittest.TestCase, MixInCheckStateHandling): def test_basics(self): @@ -1841,7 +1868,7 @@ (chars, size) = codecs.getdecoder(encoding)(b) self.assertEqual(chars, s, "encoding=%r" % encoding) - if encoding not in broken_unicode_with_streams: + if encoding not in broken_unicode_with_stateful: # check stream reader/writer q = Queue(b"") writer = codecs.getwriter(encoding)(q) @@ -1859,7 +1886,7 @@ decodedresult += reader.read() self.assertEqual(decodedresult, s, "encoding=%r" % encoding) - if encoding not in broken_incremental_coders: + if encoding not in broken_unicode_with_stateful: # check incremental decoder/encoder and iterencode()/iterdecode() try: encoder = codecs.getincrementalencoder(encoding)() @@ -1908,7 +1935,7 @@ from _testcapi import codec_incrementalencoder, codec_incrementaldecoder s = "abc123" # all codecs should be able to encode these for encoding in all_unicode_encodings: - if encoding not in broken_incremental_coders: + if encoding not in broken_unicode_with_stateful: # check incremental decoder/encoder (fetched via the C API) try: cencoder = codec_incrementalencoder(encoding) @@ -1948,7 +1975,7 @@ for encoding in all_unicode_encodings: if encoding == "idna": # FIXME: See SF bug #1163178 continue - if encoding in broken_unicode_with_streams: + if encoding in broken_unicode_with_stateful: continue reader = codecs.getreader(encoding)(io.BytesIO(s.encode(encoding))) for t in range(5): @@ -1981,7 +2008,7 @@ # Check that getstate() and setstate() handle the state properly u = "abc123" for encoding in all_unicode_encodings: - if encoding not in broken_incremental_coders: + if encoding not in broken_unicode_with_stateful: self.check_state_handling_decode(encoding, u, u.encode(encoding)) self.check_state_handling_encode(encoding, u, u.encode(encoding)) @@ -2185,6 +2212,7 @@ f = io.BytesIO(b"\xc3\xbc") with codecs.EncodedFile(f, "latin-1", "utf-8") as ef: self.assertEqual(ef.read(), b"\xfc") + self.assertTrue(f.closed) def test_streamreaderwriter(self): f = io.BytesIO(b"\xc3\xbc") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1441,6 +1441,10 @@ Documentation ------------- +- Issue #19548: Update the codecs module documentation to better cover the + distinction between text encodings and other codecs, together with other + clarifications. Patch by Martin Panter. + - Issue #22394: Doc/Makefile now supports ``make venv PYTHON=../python`` to create a venv for generating the documentation, e.g., ``make html PYTHON=venv/bin/python3``. @@ -1477,6 +1481,10 @@ Tests ----- +- Issue #19548: Added some additional checks to test_codecs to ensure that + statements in the updated documentation remain accurate. Patch by Martin + Panter. + - Issue #22838: All test_re tests now work with unittest test discovery. - Issue #22173: Update lib2to3 tests to use unittest test discovery. diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c --- a/Modules/_codecsmodule.c +++ b/Modules/_codecsmodule.c @@ -54,9 +54,9 @@ "register(search_function)\n\ \n\ Register a codec search function. Search functions are expected to take\n\ -one argument, the encoding name in all lower case letters, and return\n\ -a tuple of functions (encoder, decoder, stream_reader, stream_writer)\n\ -(or a CodecInfo object)."); +one argument, the encoding name in all lower case letters, and either\n\ +return None, or a tuple of functions (encoder, decoder, stream_reader,\n\ +stream_writer) (or a CodecInfo object)."); static PyObject *codec_register(PyObject *self, PyObject *search_function) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 15:38:52 2015 From: python-checkins at python.org (nick.coghlan) Date: Tue, 06 Jan 2015 14:38:52 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgMTk1NDg6?= =?utf-8?q?_update_codecs_module_documentation?= Message-ID: <20150106143716.22405.91104@psf.io> https://hg.python.org/cpython/rev/0646eee8296a changeset: 94053:0646eee8296a branch: 3.4 parent: 94050:7f82f50fdad0 user: Nick Coghlan date: Wed Jan 07 00:22:00 2015 +1000 summary: Issue 19548: update codecs module documentation - clarified the distinction between text encodings and other codecs - clarified relationship with builtin open and the io module - consolidated documentation of error handlers into one section - clarified type constraints of some behaviours - added tests for some of the new statements in the docs files: Doc/glossary.rst | 5 +- Doc/library/codecs.rst | 648 +++++++++++++------------ Doc/library/functions.rst | 8 +- Doc/library/stdtypes.rst | 4 +- Doc/library/tarfile.rst | 2 +- Lib/codecs.py | 71 +- Lib/test/test_codecs.py | 46 +- Misc/NEWS | 8 + Modules/_codecsmodule.c | 6 +- 9 files changed, 426 insertions(+), 372 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -820,10 +820,13 @@ :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences include :data:`sys.float_info` and the return value of :func:`os.stat`. + text encoding + A codec which encodes Unicode strings to bytes. + text file A :term:`file object` able to read and write :class:`str` objects. Often, a text file actually accesses a byte-oriented datastream - and handles the text encoding automatically. + and handles the :term:`text encoding` automatically. .. seealso:: A :term:`binary file` reads and write :class:`bytes` objects. diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -17,10 +17,17 @@ pair: stackable; streams This module defines base classes for standard Python codecs (encoders and -decoders) and provides access to the internal Python codec registry which -manages the codec and error handling lookup process. +decoders) and provides access to the internal Python codec registry, which +manages the codec and error handling lookup process. Most standard codecs +are :term:`text encodings `, which encode text to bytes, +but there are also codecs provided that encode text to text, and bytes to +bytes. Custom codecs may encode and decode between arbitrary types, but some +module features are restricted to use specifically with +:term:`text encodings `, or with codecs that encode to +:class:`bytes`. -It defines the following functions: +The module defines the following functions for encoding and decoding with +any codec: .. function:: encode(obj, [encoding[, errors]]) @@ -28,7 +35,7 @@ encoding is ``utf-8``. *Errors* may be given to set the desired error handling scheme. The - default error handler is ``strict`` meaning that encoding errors raise + default error handler is ``'strict'`` meaning that encoding errors raise :exc:`ValueError` (or a more codec specific subclass, such as :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. @@ -39,90 +46,63 @@ encoding is ``utf-8``. *Errors* may be given to set the desired error handling scheme. The - default error handler is ``strict`` meaning that decoding errors raise + default error handler is ``'strict'`` meaning that decoding errors raise :exc:`ValueError` (or a more codec specific subclass, such as :exc:`UnicodeDecodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. -.. function:: register(search_function) - - Register a codec search function. Search functions are expected to take one - argument, the encoding name in all lower case letters, and return a - :class:`CodecInfo` object having the following attributes: - - * ``name`` The name of the encoding; - - * ``encode`` The stateless encoding function; - - * ``decode`` The stateless decoding function; - - * ``incrementalencoder`` An incremental encoder class or factory function; - - * ``incrementaldecoder`` An incremental decoder class or factory function; - - * ``streamwriter`` A stream writer class or factory function; - - * ``streamreader`` A stream reader class or factory function. - - The various functions or classes take the following arguments: - - *encode* and *decode*: These must be functions or methods which have the same - interface as the :meth:`~Codec.encode`/:meth:`~Codec.decode` methods of Codec - instances (see :ref:`Codec Interface `). The functions/methods - are expected to work in a stateless mode. - - *incrementalencoder* and *incrementaldecoder*: These have to be factory - functions providing the following interface: - - ``factory(errors='strict')`` - - The factory functions must return objects providing the interfaces defined by - the base classes :class:`IncrementalEncoder` and :class:`IncrementalDecoder`, - respectively. Incremental codecs can maintain state. - - *streamreader* and *streamwriter*: These have to be factory functions providing - the following interface: - - ``factory(stream, errors='strict')`` - - The factory functions must return objects providing the interfaces defined by - the base classes :class:`StreamReader` and :class:`StreamWriter`, respectively. - Stream codecs can maintain state. - - Possible values for errors are - - * ``'strict'``: raise an exception in case of an encoding error - * ``'replace'``: replace malformed data with a suitable replacement marker, - such as ``'?'`` or ``'\ufffd'`` - * ``'ignore'``: ignore malformed data and continue without further notice - * ``'xmlcharrefreplace'``: replace with the appropriate XML character - reference (for encoding only) - * ``'backslashreplace'``: replace with backslashed escape sequences (for - encoding only) - * ``'surrogateescape'``: on decoding, replace with code points in the Unicode - Private Use Area ranging from U+DC80 to U+DCFF. These private code - points will then be turned back into the same bytes when the - ``surrogateescape`` error handler is used when encoding the data. - (See :pep:`383` for more.) - - as well as any other error handling name defined via :func:`register_error`. - - In case a search function cannot find a given encoding, it should return - ``None``. - +The full details for each codec can also be looked up directly: .. function:: lookup(encoding) Looks up the codec info in the Python codec registry and returns a - :class:`CodecInfo` object as defined above. + :class:`CodecInfo` object as defined below. Encodings are first looked up in the registry's cache. If not found, the list of registered search functions is scanned. If no :class:`CodecInfo` object is found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object is stored in the cache and returned to the caller. -To simplify access to the various codecs, the module provides these additional -functions which use :func:`lookup` for the codec lookup: +.. class:: CodecInfo(encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None) + + Codec details when looking up the codec registry. The constructor + arguments are stored in attributes of the same name: + + + .. attribute:: name + + The name of the encoding. + + + .. attribute:: encode + decode + + The stateless encoding and decoding functions. These must be + functions or methods which have the same interface as + the :meth:`~Codec.encode` and :meth:`~Codec.decode` methods of Codec + instances (see :ref:`Codec Interface `). + The functions or methods are expected to work in a stateless mode. + + + .. attribute:: incrementalencoder + incrementaldecoder + + Incremental encoder and decoder classes or factory functions. + These have to provide the interface defined by the base classes + :class:`IncrementalEncoder` and :class:`IncrementalDecoder`, + respectively. Incremental codecs can maintain state. + + + .. attribute:: streamwriter + streamreader + + Stream writer and reader classes or factory functions. These have to + provide the interface defined by the base classes + :class:`StreamWriter` and :class:`StreamReader`, respectively. + Stream codecs can maintain state. + +To simplify access to the various codec components, the module provides +these additional functions which use :func:`lookup` for the codec lookup: .. function:: getencoder(encoding) @@ -172,90 +152,43 @@ Raises a :exc:`LookupError` in case the encoding cannot be found. +Custom codecs are made available by registering a suitable codec search +function: -.. function:: register_error(name, error_handler) +.. function:: register(search_function) - Register the error handling function *error_handler* under the name *name*. - *error_handler* will be called during encoding and decoding in case of an error, - when *name* is specified as the errors parameter. - - For encoding *error_handler* will be called with a :exc:`UnicodeEncodeError` - instance, which contains information about the location of the error. The - error handler must either raise this or a different exception or return a - tuple with a replacement for the unencodable part of the input and a position - where encoding should continue. The replacement may be either :class:`str` or - :class:`bytes`. If the replacement is bytes, the encoder will simply copy - them into the output buffer. If the replacement is a string, the encoder will - encode the replacement. Encoding continues on original input at the - specified position. Negative position values will be treated as being - relative to the end of the input string. If the resulting position is out of - bound an :exc:`IndexError` will be raised. - - Decoding and translating works similar, except :exc:`UnicodeDecodeError` or - :exc:`UnicodeTranslateError` will be passed to the handler and that the - replacement from the error handler will be put into the output directly. - - -.. function:: lookup_error(name) - - Return the error handler previously registered under the name *name*. - - Raises a :exc:`LookupError` in case the handler cannot be found. - - -.. function:: strict_errors(exception) - - Implements the ``strict`` error handling: each encoding or decoding error - raises a :exc:`UnicodeError`. - - -.. function:: replace_errors(exception) - - Implements the ``replace`` error handling: malformed data is replaced with a - suitable replacement character such as ``'?'`` in bytestrings and - ``'\ufffd'`` in Unicode strings. - - -.. function:: ignore_errors(exception) - - Implements the ``ignore`` error handling: malformed data is ignored and - encoding or decoding is continued without further notice. - - -.. function:: xmlcharrefreplace_errors(exception) - - Implements the ``xmlcharrefreplace`` error handling (for encoding only): the - unencodable character is replaced by an appropriate XML character reference. - - -.. function:: backslashreplace_errors(exception) - - Implements the ``backslashreplace`` error handling (for encoding only): the - unencodable character is replaced by a backslashed escape sequence. - -To simplify working with encoded files or stream, the module also defines these -utility functions: - - -.. function:: open(filename, mode[, encoding[, errors[, buffering]]]) - - Open an encoded file using the given *mode* and return a wrapped version - providing transparent encoding/decoding. The default file mode is ``'r'`` - meaning to open the file in read mode. + Register a codec search function. Search functions are expected to take one + argument, being the encoding name in all lower case letters, and return a + :class:`CodecInfo` object. In case a search function cannot find + a given encoding, it should return ``None``. .. note:: - The wrapped version's methods will accept and return strings only. Bytes - arguments will be rejected. + Search function registration is not currently reversible, + which may cause problems in some cases, such as unit testing or + module reloading. + +While the builtin :func:`open` and the associated :mod:`io` module are the +recommended approach for working with encoded text files, this module +provides additional utility functions and classes that allow the use of a +wider range of codecs when working with binary files: + +.. function:: open(filename, mode='r', encoding=None, errors='strict', buffering=1) + + Open an encoded file using the given *mode* and return an instance of + :class:`StreamReaderWriter`, providing transparent encoding/decoding. + The default file mode is ``'r'``, meaning to open the file in read mode. .. note:: - Files are always opened in binary mode, even if no binary mode was - specified. This is done to avoid data loss due to encodings using 8-bit - values. This means that no automatic conversion of ``b'\n'`` is done - on reading and writing. + Underlying encoded files are always opened in binary mode. + No automatic conversion of ``'\n'`` is done on reading and writing. + The *mode* argument may be any binary mode acceptable to the built-in + :func:`open` function; the ``'b'`` is automatically added. *encoding* specifies the encoding which is to be used for the file. + Any encoding that encodes to and decodes from bytes is allowed, and + the data types supported by the file methods depend on the codec used. *errors* may be given to define the error handling. It defaults to ``'strict'`` which causes a :exc:`ValueError` to be raised in case an encoding error occurs. @@ -266,12 +199,15 @@ .. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict') - Return a wrapped version of file which provides transparent encoding - translation. + Return a :class:`StreamRecoder` instance, a wrapped version of *file* + which provides transparent transcoding. The original file is closed + when the wrapped version is closed. - Bytes written to the wrapped file are interpreted according to the given - *data_encoding* and then written to the original file as bytes using the - *file_encoding*. + Data written to the wrapped file is decoded according to the given + *data_encoding* and then written to the original file as bytes using + *file_encoding*. Bytes read from the original file are decoded + according to *file_encoding*, and the result is encoded + using *data_encoding*. If *file_encoding* is not given, it defaults to *data_encoding*. @@ -283,14 +219,16 @@ .. function:: iterencode(iterator, encoding, errors='strict', **kwargs) Uses an incremental encoder to iteratively encode the input provided by - *iterator*. This function is a :term:`generator`. *errors* (as well as any + *iterator*. This function is a :term:`generator`. + The *errors* argument (as well as any other keyword argument) is passed through to the incremental encoder. .. function:: iterdecode(iterator, encoding, errors='strict', **kwargs) Uses an incremental decoder to iteratively decode the input provided by - *iterator*. This function is a :term:`generator`. *errors* (as well as any + *iterator*. This function is a :term:`generator`. + The *errors* argument (as well as any other keyword argument) is passed through to the incremental decoder. @@ -309,9 +247,10 @@ BOM_UTF32_BE BOM_UTF32_LE - These constants define various encodings of the Unicode byte order mark (BOM) - used in UTF-16 and UTF-32 data streams to indicate the byte order used in the - stream or file and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either + These constants define various byte sequences, + being Unicode byte order marks (BOMs) for several encodings. They are + used in UTF-16 and UTF-32 data streams to indicate the byte order used, + and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`, :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for @@ -325,20 +264,25 @@ ------------------ The :mod:`codecs` module defines a set of base classes which define the -interface and can also be used to easily write your own codecs for use in -Python. +interfaces for working with codec objects, and can also be used as the basis +for custom codec implementations. Each codec has to define four interfaces to make it usable as codec in Python: stateless encoder, stateless decoder, stream reader and stream writer. The stream reader and writers typically reuse the stateless encoder/decoder to -implement the file protocols. +implement the file protocols. Codec authors also need to define how the +codec will handle encoding and decoding errors. -The :class:`Codec` class defines the interface for stateless encoders/decoders. -To simplify and standardize error handling, the :meth:`~Codec.encode` and -:meth:`~Codec.decode` methods may implement different error handling schemes by -providing the *errors* string argument. The following string values are defined -and implemented by all standard Python codecs: +.. _error-handlers: + +Error Handlers +^^^^^^^^^^^^^^ + +To simplify and standardize error handling, +codecs may implement different error handling schemes by +accepting the *errors* string argument. The following string values are +defined and implemented by all standard Python codecs: .. tabularcolumns:: |l|L| @@ -346,36 +290,52 @@ | Value | Meaning | +=========================+===============================================+ | ``'strict'`` | Raise :exc:`UnicodeError` (or a subclass); | -| | this is the default. | +| | this is the default. Implemented in | +| | :func:`strict_errors`. | +-------------------------+-----------------------------------------------+ -| ``'ignore'`` | Ignore the character and continue with the | -| | next. | +| ``'ignore'`` | Ignore the malformed data and continue | +| | without further notice. Implemented in | +| | :func:`ignore_errors`. | +-------------------------+-----------------------------------------------+ + +The following error handlers are only applicable to +:term:`text encodings `: + ++-------------------------+-----------------------------------------------+ +| Value | Meaning | ++=========================+===============================================+ | ``'replace'`` | Replace with a suitable replacement | -| | character; Python will use the official | -| | U+FFFD REPLACEMENT CHARACTER for the built-in | -| | Unicode codecs on decoding and '?' on | -| | encoding. | +| | marker; Python will use the official | +| | ``U+FFFD`` REPLACEMENT CHARACTER for the | +| | built-in codecs on decoding, and '?' on | +| | encoding. Implemented in | +| | :func:`replace_errors`. | +-------------------------+-----------------------------------------------+ | ``'xmlcharrefreplace'`` | Replace with the appropriate XML character | -| | reference (only for encoding). | +| | reference (only for encoding). Implemented | +| | in :func:`xmlcharrefreplace_errors`. | +-------------------------+-----------------------------------------------+ | ``'backslashreplace'`` | Replace with backslashed escape sequences | -| | (only for encoding). | +| | (only for encoding). Implemented in | +| | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ -| ``'surrogateescape'`` | Replace byte with surrogate U+DCxx, as defined| -| | in :pep:`383`. | +| ``'surrogateescape'`` | On decoding, replace byte with individual | +| | surrogate code ranging from ``U+DC80`` to | +| | ``U+DCFF``. This code will then be turned | +| | back into the same byte when the | +| | ``'surrogateescape'`` error handler is used | +| | when encoding the data. (See :pep:`383` for | +| | more.) | +-------------------------+-----------------------------------------------+ -In addition, the following error handlers are specific to Unicode encoding -schemes: +In addition, the following error handler is specific to the given codecs: +-------------------+------------------------+-------------------------------------------+ -| Value | Codec | Meaning | +| Value | Codecs | Meaning | +===================+========================+===========================================+ |``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding of surrogate | -| | utf-16-be, utf-16-le, | codes in all the Unicode encoding schemes.| -| | utf-32-be, utf-32-le | | +| | utf-16-be, utf-16-le, | codes. These codecs normally treat the | +| | utf-32-be, utf-32-le | presence of surrogates as an error. | +-------------------+------------------------+-------------------------------------------+ .. versionadded:: 3.1 @@ -384,26 +344,96 @@ .. versionchanged:: 3.4 The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs. -The set of allowed values can be extended via :meth:`register_error`. +The set of allowed values can be extended by registering a new named error +handler: + +.. function:: register_error(name, error_handler) + + Register the error handling function *error_handler* under the name *name*. + The *error_handler* argument will be called during encoding and decoding + in case of an error, when *name* is specified as the errors parameter. + + For encoding, *error_handler* will be called with a :exc:`UnicodeEncodeError` + instance, which contains information about the location of the error. The + error handler must either raise this or a different exception, or return a + tuple with a replacement for the unencodable part of the input and a position + where encoding should continue. The replacement may be either :class:`str` or + :class:`bytes`. If the replacement is bytes, the encoder will simply copy + them into the output buffer. If the replacement is a string, the encoder will + encode the replacement. Encoding continues on original input at the + specified position. Negative position values will be treated as being + relative to the end of the input string. If the resulting position is out of + bound an :exc:`IndexError` will be raised. + + Decoding and translating works similarly, except :exc:`UnicodeDecodeError` or + :exc:`UnicodeTranslateError` will be passed to the handler and that the + replacement from the error handler will be put into the output directly. + + +Previously registered error handlers (including the standard error handlers) +can be looked up by name: + +.. function:: lookup_error(name) + + Return the error handler previously registered under the name *name*. + + Raises a :exc:`LookupError` in case the handler cannot be found. + +The following standard error handlers are also made available as module level +functions: + +.. function:: strict_errors(exception) + + Implements the ``'strict'`` error handling: each encoding or + decoding error raises a :exc:`UnicodeError`. + + +.. function:: replace_errors(exception) + + Implements the ``'replace'`` error handling (for :term:`text encodings + ` only): substitutes ``'?'`` for encoding errors + (to be encoded by the codec), and ``'\ufffd'`` (the Unicode replacement + character, ``'?'``) for decoding errors. + + +.. function:: ignore_errors(exception) + + Implements the ``'ignore'`` error handling: malformed data is ignored and + encoding or decoding is continued without further notice. + + +.. function:: xmlcharrefreplace_errors(exception) + + Implements the ``'xmlcharrefreplace'`` error handling (for encoding with + :term:`text encodings ` only): the + unencodable character is replaced by an appropriate XML character reference. + + +.. function:: backslashreplace_errors(exception) + + Implements the ``'backslashreplace'`` error handling (for encoding with + :term:`text encodings ` only): the + unencodable character is replaced by a backslashed escape sequence. .. _codec-objects: -Codec Objects -^^^^^^^^^^^^^ +Stateless Encoding and Decoding +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The :class:`Codec` class defines these methods which also define the function -interfaces of the stateless encoder and decoder: +The base :class:`Codec` class defines these methods which also define the +function interfaces of the stateless encoder and decoder: .. method:: Codec.encode(input[, errors]) Encodes the object *input* and returns a tuple (output object, length consumed). - Encoding converts a string object to a bytes object using a particular + For instance, :term:`text encoding` converts + a string object to a bytes object using a particular character set encoding (e.g., ``cp1252`` or ``iso-8859-1``). - *errors* defines the error handling to apply. It defaults to ``'strict'`` - handling. + The *errors* argument defines the error handling to apply. + It defaults to ``'strict'`` handling. The method may not store state in the :class:`Codec` instance. Use :class:`StreamCodec` for codecs which have to keep state in order to make @@ -416,14 +446,16 @@ .. method:: Codec.decode(input[, errors]) Decodes the object *input* and returns a tuple (output object, length - consumed). Decoding converts a bytes object encoded using a particular + consumed). For instance, for a :term:`text encoding`, decoding converts + a bytes object encoded using a particular character set encoding to a string object. - *input* must be a bytes object or one which provides the read-only character + For text encodings and bytes-to-bytes codecs, + *input* must be a bytes object or one which provides the read-only buffer interface -- for example, buffer objects and memory mapped files. - *errors* defines the error handling to apply. It defaults to ``'strict'`` - handling. + The *errors* argument defines the error handling to apply. + It defaults to ``'strict'`` handling. The method may not store state in the :class:`Codec` instance. Use :class:`StreamCodec` for codecs which have to keep state in order to make @@ -432,6 +464,10 @@ The decoder must be able to handle zero length input and return an empty object of the output object type in this situation. + +Incremental Encoding and Decoding +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide the basic interface for incremental encoding and decoding. Encoding/decoding the input isn't done with one call to the stateless encoder/decoder function, but @@ -449,14 +485,14 @@ .. _incremental-encoder-objects: IncrementalEncoder Objects -^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~ The :class:`IncrementalEncoder` class is used for encoding an input in multiple steps. It defines the following methods which every incremental encoder must define in order to be compatible with the Python codec registry. -.. class:: IncrementalEncoder([errors]) +.. class:: IncrementalEncoder(errors='strict') Constructor for an :class:`IncrementalEncoder` instance. @@ -465,26 +501,14 @@ the Python codec registry. The :class:`IncrementalEncoder` may implement different error handling schemes - by providing the *errors* keyword argument. These parameters are predefined: - - * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default. - - * ``'ignore'`` Ignore the character and continue with the next. - - * ``'replace'`` Replace with a suitable replacement character - - * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference - - * ``'backslashreplace'`` Replace with backslashed escape sequences. + by providing the *errors* keyword argument. See :ref:`error-handlers` for + possible values. The *errors* argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`IncrementalEncoder` object. - The set of allowed values for the *errors* argument can be extended with - :func:`register_error`. - .. method:: encode(object[, final]) @@ -496,7 +520,8 @@ .. method:: reset() Reset the encoder to the initial state. The output is discarded: call - ``.encode('', final=True)`` to reset the encoder and to get the output. + ``.encode(object, final=True)``, passing an empty byte or text string + if necessary, to reset the encoder and to get the output. .. method:: IncrementalEncoder.getstate() @@ -517,14 +542,14 @@ .. _incremental-decoder-objects: IncrementalDecoder Objects -^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~ The :class:`IncrementalDecoder` class is used for decoding an input in multiple steps. It defines the following methods which every incremental decoder must define in order to be compatible with the Python codec registry. -.. class:: IncrementalDecoder([errors]) +.. class:: IncrementalDecoder(errors='strict') Constructor for an :class:`IncrementalDecoder` instance. @@ -533,22 +558,14 @@ the Python codec registry. The :class:`IncrementalDecoder` may implement different error handling schemes - by providing the *errors* keyword argument. These parameters are predefined: - - * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default. - - * ``'ignore'`` Ignore the character and continue with the next. - - * ``'replace'`` Replace with a suitable replacement character. + by providing the *errors* keyword argument. See :ref:`error-handlers` for + possible values. The *errors* argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`IncrementalDecoder` object. - The set of allowed values for the *errors* argument can be extended with - :func:`register_error`. - .. method:: decode(object[, final]) @@ -587,6 +604,10 @@ returned by :meth:`getstate`. +Stream Encoding and Decoding +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + The :class:`StreamWriter` and :class:`StreamReader` classes provide generic working interfaces which can be used to implement new encoding submodules very easily. See :mod:`encodings.utf_8` for an example of how this is done. @@ -595,14 +616,14 @@ .. _stream-writer-objects: StreamWriter Objects -^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~ The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the following methods which every stream writer must define in order to be compatible with the Python codec registry. -.. class:: StreamWriter(stream[, errors]) +.. class:: StreamWriter(stream, errors='strict') Constructor for a :class:`StreamWriter` instance. @@ -610,29 +631,17 @@ additional keyword arguments, but only the ones defined here are used by the Python codec registry. - *stream* must be a file-like object open for writing binary data. + The *stream* argument must be a file-like object open for writing + text or binary data, as appropriate for the specific codec. The :class:`StreamWriter` may implement different error handling schemes by - providing the *errors* keyword argument. These parameters are predefined: - - * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default. - - * ``'ignore'`` Ignore the character and continue with the next. - - * ``'replace'`` Replace with a suitable replacement character - - * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference - - * ``'backslashreplace'`` Replace with backslashed escape sequences. + providing the *errors* keyword argument. See :ref:`error-handlers` for + the standard error handlers the underlying stream codec may support. The *errors* argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`StreamWriter` object. - The set of allowed values for the *errors* argument can be extended with - :func:`register_error`. - - .. method:: write(object) Writes the object's contents encoded to the stream. @@ -641,7 +650,8 @@ .. method:: writelines(list) Writes the concatenated list of strings to the stream (possibly by reusing - the :meth:`write` method). + the :meth:`write` method). The standard bytes-to-bytes codecs + do not support this method. .. method:: reset() @@ -660,14 +670,14 @@ .. _stream-reader-objects: StreamReader Objects -^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~ The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the following methods which every stream reader must define in order to be compatible with the Python codec registry. -.. class:: StreamReader(stream[, errors]) +.. class:: StreamReader(stream, errors='strict') Constructor for a :class:`StreamReader` instance. @@ -675,16 +685,12 @@ additional keyword arguments, but only the ones defined here are used by the Python codec registry. - *stream* must be a file-like object open for reading (binary) data. + The *stream* argument must be a file-like object open for reading + text or binary data, as appropriate for the specific codec. The :class:`StreamReader` may implement different error handling schemes by - providing the *errors* keyword argument. These parameters are defined: - - * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default. - - * ``'ignore'`` Ignore the character and continue with the next. - - * ``'replace'`` Replace with a suitable replacement character. + providing the *errors* keyword argument. See :ref:`error-handlers` for + the standard error handlers the underlying stream codec may support. The *errors* argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error @@ -698,17 +704,20 @@ Decodes data from the stream and returns the resulting object. - *chars* indicates the number of characters to read from the - stream. :func:`read` will never return more than *chars* characters, but - it might return less, if there are not enough characters available. + The *chars* argument indicates the number of decoded + code points or bytes to return. The :func:`read` method will + never return more data than requested, but it might return less, + if there is not enough available. - *size* indicates the approximate maximum number of bytes to read from the - stream for decoding purposes. The decoder can modify this setting as + The *size* argument indicates the approximate maximum + number of encoded bytes or code points to read + for decoding. The decoder can modify this setting as appropriate. The default value -1 indicates to read and decode as much as - possible. *size* is intended to prevent having to decode huge files in - one step. + possible. This parameter is intended to + prevent having to decode huge files in one step. - *firstline* indicates that it would be sufficient to only return the first + The *firstline* flag indicates that + it would be sufficient to only return the first line, if there are decoding errors on later lines. The method should use a greedy read strategy meaning that it should read @@ -751,17 +760,13 @@ In addition to the above methods, the :class:`StreamReader` must also inherit all other methods and attributes from the underlying stream. -The next two base classes are included for convenience. They are not needed by -the codec registry, but may provide useful in practice. - - .. _stream-reader-writer: StreamReaderWriter Objects -^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~ -The :class:`StreamReaderWriter` allows wrapping streams which work in both read -and write modes. +The :class:`StreamReaderWriter` is a convenience class that allows wrapping +streams which work in both read and write modes. The design is such that one can use the factory functions returned by the :func:`lookup` function to construct the instance. @@ -782,9 +787,9 @@ .. _stream-recoder-objects: StreamRecoder Objects -^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~ -The :class:`StreamRecoder` provide a frontend - backend view of encoding data +The :class:`StreamRecoder` translates data from one encoding to another, which is sometimes useful when dealing with different encoding environments. The design is such that one can use the factory functions returned by the @@ -794,22 +799,20 @@ .. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors) Creates a :class:`StreamRecoder` instance which implements a two-way conversion: - *encode* and *decode* work on the frontend (the input to :meth:`read` and output - of :meth:`write`) while *Reader* and *Writer* work on the backend (reading and - writing to the stream). + *encode* and *decode* work on the frontend?? the data visible to + code calling :meth:`read` and :meth:`write`, while *Reader* and *Writer* + work on the backend?? the data in *stream*. - You can use these objects to do transparent direct recodings from e.g. Latin-1 + You can use these objects to do transparent transcodings from e.g. Latin-1 to UTF-8 and back. - *stream* must be a file-like object. + The *stream* argument must be a file-like object. - *encode*, *decode* must adhere to the :class:`Codec` interface. *Reader*, + The *encode* and *decode* arguments must + adhere to the :class:`Codec` interface. *Reader* and *Writer* must be factory functions or classes providing objects of the :class:`StreamReader` and :class:`StreamWriter` interface respectively. - *encode* and *decode* are needed for the frontend translation, *Reader* and - *Writer* for the backend translation. - Error handling is done in the same way as defined for the stream readers and writers. @@ -824,20 +827,23 @@ Encodings and Unicode --------------------- -Strings are stored internally as sequences of codepoints in range ``0 - 10FFFF`` -(see :pep:`393` for more details about the implementation). -Once a string object is used outside of CPU and memory, CPU endianness -and how these arrays are stored as bytes become an issue. Transforming a -string object into a sequence of bytes is called encoding and recreating the -string object from the sequence of bytes is known as decoding. There are many -different methods for how this transformation can be done (these methods are -also called encodings). The simplest method is to map the codepoints 0-255 to -the bytes ``0x0``-``0xff``. This means that a string object that contains -codepoints above ``U+00FF`` can't be encoded with this method (which is called -``'latin-1'`` or ``'iso-8859-1'``). :func:`str.encode` will raise a -:exc:`UnicodeEncodeError` that looks like this: ``UnicodeEncodeError: 'latin-1' -codec can't encode character '\u1234' in position 3: ordinal not in -range(256)``. +Strings are stored internally as sequences of codepoints in +range ``0x0``-``0x10FFFF``. (See :pep:`393` for +more details about the implementation.) +Once a string object is used outside of CPU and memory, endianness +and how these arrays are stored as bytes become an issue. As with other +codecs, serialising a string into a sequence of bytes is known as *encoding*, +and recreating the string from the sequence of bytes is known as *decoding*. +There are a variety of different text serialisation codecs, which are +collectivity referred to as :term:`text encodings `. + +The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps +the codepoints 0-255 to the bytes ``0x0``-``0xff``, which means that a string +object that contains codepoints above ``U+00FF`` can't be encoded with this +codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks +like the following (although the details of the error message may differ): +``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in +position 3: ordinal not in range(256)``. There's another group of encodings (the so called charmap encodings) that choose a different subset of all Unicode code points and how these codepoints are @@ -1184,7 +1190,8 @@ .. versionchanged:: 3.4 The utf-16\* and utf-32\* encoders no longer allow surrogate code points - (U+D800--U+DFFF) to be encoded. The utf-32\* decoders no longer decode + (``U+D800``--``U+DFFF``) to be encoded. + The utf-32\* decoders no longer decode byte sequences that correspond to surrogate code points. @@ -1212,7 +1219,9 @@ +====================+=========+===========================+ | idna | | Implements :rfc:`3490`, | | | | see also | -| | | :mod:`encodings.idna` | +| | | :mod:`encodings.idna`. | +| | | Only ``errors='strict'`` | +| | | is supported. | +--------------------+---------+---------------------------+ | mbcs | dbcs | Windows only: Encode | | | | operand according to the | @@ -1220,31 +1229,44 @@ +--------------------+---------+---------------------------+ | palmos | | Encoding of PalmOS 3.5 | +--------------------+---------+---------------------------+ -| punycode | | Implements :rfc:`3492` | +| punycode | | Implements :rfc:`3492`. | +| | | Stateful codecs are not | +| | | supported. | +--------------------+---------+---------------------------+ -| raw_unicode_escape | | Produce a string that is | -| | | suitable as raw Unicode | -| | | literal in Python source | -| | | code | +| raw_unicode_escape | | Latin-1 encoding with | +| | | ``\uXXXX`` and | +| | | ``\UXXXXXXXX`` for other | +| | | code points. Existing | +| | | backslashes are not | +| | | escaped in any way. | +| | | It is used in the Python | +| | | pickle protocol. | +--------------------+---------+---------------------------+ | undefined | | Raise an exception for | -| | | all conversions. Can be | -| | | used as the system | -| | | encoding if no automatic | -| | | coercion between byte and | -| | | Unicode strings is | -| | | desired. | +| | | all conversions, even | +| | | empty strings. The error | +| | | handler is ignored. | +--------------------+---------+---------------------------+ -| unicode_escape | | Produce a string that is | -| | | suitable as Unicode | -| | | literal in Python source | -| | | code | +| unicode_escape | | Encoding suitable as the | +| | | contents of a Unicode | +| | | literal in ASCII-encoded | +| | | Python source code, | +| | | except that quotes are | +| | | not escaped. Decodes from | +| | | Latin-1 source code. | +| | | Beware that Python source | +| | | code actually uses UTF-8 | +| | | by default. | +--------------------+---------+---------------------------+ | unicode_internal | | Return the internal | | | | representation of the | -| | | operand | +| | | operand. Stateful codecs | +| | | are not supported. | | | | | | | | .. deprecated:: 3.3 | +| | | This representation is | +| | | obsoleted by | +| | | :pep:`393`. | +--------------------+---------+---------------------------+ .. _binary-transforms: @@ -1253,7 +1275,8 @@ ^^^^^^^^^^^^^^^^^ The following codecs provide binary transforms: :term:`bytes-like object` -to :class:`bytes` mappings. +to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode` +(which only produces :class:`str` output). .. tabularcolumns:: |l|L|L|L| @@ -1308,7 +1331,8 @@ ^^^^^^^^^^^^^^^ The following codec provides a text transform: a :class:`str` to :class:`str` -mapping. +mapping. It is not supported by :meth:`str.encode` (which only produces +:class:`bytes` output). .. tabularcolumns:: |l|l|L| diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -939,15 +939,17 @@ *encoding* is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever :func:`locale.getpreferredencoding` returns), but any - encoding supported by Python can be used. See the :mod:`codecs` module for + :term:`text encoding` supported by Python + can be used. See the :mod:`codecs` module for the list of supported encodings. *errors* is an optional string that specifies how encoding and decoding errors are to be handled--this cannot be used in binary mode. - A variety of standard error handlers are available, though any + A variety of standard error handlers are available + (listed under :ref:`error-handlers`), though any error handling name that has been registered with :func:`codecs.register_error` is also valid. The standard names - are: + include: * ``'strict'`` to raise a :exc:`ValueError` exception if there is an encoding error. The default value of ``None`` has the same diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1512,7 +1512,7 @@ a :exc:`UnicodeError`. Other possible values are ``'ignore'``, ``'replace'``, ``'xmlcharrefreplace'``, ``'backslashreplace'`` and any other name registered via - :func:`codecs.register_error`, see section :ref:`codec-base-classes`. For a + :func:`codecs.register_error`, see section :ref:`error-handlers`. For a list of possible encodings, see section :ref:`standard-encodings`. .. versionchanged:: 3.1 @@ -2384,7 +2384,7 @@ error handling scheme. The default for *errors* is ``'strict'``, meaning that encoding errors raise a :exc:`UnicodeError`. Other possible values are ``'ignore'``, ``'replace'`` and any other name registered via - :func:`codecs.register_error`, see section :ref:`codec-base-classes`. For a + :func:`codecs.register_error`, see section :ref:`error-handlers`. For a list of possible encodings, see section :ref:`standard-encodings`. .. note:: diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -794,7 +794,7 @@ appropriately, this conversion may fail. The *errors* argument defines how characters are treated that cannot be -converted. Possible values are listed in section :ref:`codec-base-classes`. +converted. Possible values are listed in section :ref:`error-handlers`. The default scheme is ``'surrogateescape'`` which Python also uses for its file system calls, see :ref:`os-filenames`. diff --git a/Lib/codecs.py b/Lib/codecs.py --- a/Lib/codecs.py +++ b/Lib/codecs.py @@ -346,8 +346,7 @@ """ Creates a StreamWriter instance. - stream must be a file-like object open for writing - (binary) data. + stream must be a file-like object open for writing. The StreamWriter may use different error handling schemes by providing the errors keyword argument. These @@ -421,8 +420,7 @@ """ Creates a StreamReader instance. - stream must be a file-like object open for reading - (binary) data. + stream must be a file-like object open for reading. The StreamReader may use different error handling schemes by providing the errors keyword argument. These @@ -450,13 +448,12 @@ """ Decodes data from the stream self.stream and returns the resulting object. - chars indicates the number of characters to read from the - stream. read() will never return more than chars - characters, but it might return less, if there are not enough - characters available. + chars indicates the number of decoded code points or bytes to + return. read() will never return more data than requested, + but it might return less, if there is not enough available. - size indicates the approximate maximum number of bytes to - read from the stream for decoding purposes. The decoder + size indicates the approximate maximum number of decoded + bytes or code points to read for decoding. The decoder can modify this setting as appropriate. The default value -1 indicates to read and decode as much as possible. size is intended to prevent having to decode huge files in one @@ -467,7 +464,7 @@ will be returned, the rest of the input will be kept until the next call to read(). - The method should use a greedy read strategy meaning that + The method should use a greedy read strategy, meaning that it should read as much data as is allowed within the definition of the encoding and the given size, e.g. if optional encoding endings or state markers are available @@ -602,7 +599,7 @@ def readlines(self, sizehint=None, keepends=True): """ Read all lines available on the input stream - and return them as list of lines. + and return them as a list. Line breaks are implemented using the codec's decoder method and are included in the list entries. @@ -750,19 +747,18 @@ class StreamRecoder: - """ StreamRecoder instances provide a frontend - backend - view of encoding data. + """ StreamRecoder instances translate data from one encoding to another. They use the complete set of APIs returned by the codecs.lookup() function to implement their task. - Data written to the stream is first decoded into an - intermediate format (which is dependent on the given codec - combination) and then written to the stream using an instance - of the provided Writer class. + Data written to the StreamRecoder is first decoded into an + intermediate format (depending on the "decode" codec) and then + written to the underlying stream using an instance of the provided + Writer class. - In the other direction, data is read from the stream using a - Reader instance and then return encoded data to the caller. + In the other direction, data is read from the underlying stream using + a Reader instance and then encoded and returned to the caller. """ # Optional attributes set by the file wrappers below @@ -774,22 +770,17 @@ """ Creates a StreamRecoder instance which implements a two-way conversion: encode and decode work on the frontend (the - input to .read() and output of .write()) while - Reader and Writer work on the backend (reading and - writing to the stream). + data visible to .read() and .write()) while Reader and Writer + work on the backend (the data in stream). - You can use these objects to do transparent direct - recodings from e.g. latin-1 to utf-8 and back. + You can use these objects to do transparent + transcodings from e.g. latin-1 to utf-8 and back. stream must be a file-like object. - encode, decode must adhere to the Codec interface, Reader, + encode and decode must adhere to the Codec interface; Reader and Writer must be factory functions or classes providing the - StreamReader, StreamWriter interface resp. - - encode and decode are needed for the frontend translation, - Reader and Writer for the backend translation. Unicode is - used as intermediate encoding. + StreamReader and StreamWriter interfaces resp. Error handling is done in the same way as defined for the StreamWriter/Readers. @@ -864,7 +855,7 @@ ### Shortcuts -def open(filename, mode='rb', encoding=None, errors='strict', buffering=1): +def open(filename, mode='r', encoding=None, errors='strict', buffering=1): """ Open an encoded file using the given mode and return a wrapped version providing transparent encoding/decoding. @@ -874,10 +865,8 @@ codecs. Output is also codec dependent and will usually be Unicode as well. - Files are always opened in binary mode, even if no binary mode - was specified. This is done to avoid data loss due to encodings - using 8-bit values. The default file mode is 'rb' meaning to - open the file in binary read mode. + Underlying encoded files are always opened in binary mode. + The default file mode is 'r', meaning to open the file in read mode. encoding specifies the encoding which is to be used for the file. @@ -913,13 +902,13 @@ """ Return a wrapped version of file which provides transparent encoding translation. - Strings written to the wrapped file are interpreted according - to the given data_encoding and then written to the original - file as string using file_encoding. The intermediate encoding + Data written to the wrapped file is decoded according + to the given data_encoding and then encoded to the underlying + file using file_encoding. The intermediate data type will usually be Unicode but depends on the specified codecs. - Strings are read from the file using file_encoding and then - passed back to the caller as string using data_encoding. + Bytes read from the file are decoded using file_encoding and then + passed back to the caller encoded using data_encoding. If file_encoding is not given, it defaults to data_encoding. diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1139,6 +1139,8 @@ # Python used to crash on this at exit because of a refcount # bug in _codecsmodule.c + self.assertTrue(f.closed) + # From RFC 3492 punycode_testcases = [ # A Arabic (Egyptian): @@ -1591,6 +1593,16 @@ self.assertEqual(encoder.encode("ample.org."), b"xn--xample-9ta.org.") self.assertEqual(encoder.encode("", True), b"") + def test_errors(self): + """Only supports "strict" error handler""" + "python.org".encode("idna", "strict") + b"python.org".decode("idna", "strict") + for errors in ("ignore", "replace", "backslashreplace", + "surrogateescape"): + self.assertRaises(Exception, "python.org".encode, "idna", errors) + self.assertRaises(Exception, + b"python.org".decode, "idna", errors) + class CodecsModuleTest(unittest.TestCase): def test_decode(self): @@ -1668,6 +1680,24 @@ for api in codecs.__all__: getattr(codecs, api) + def test_open(self): + self.addCleanup(support.unlink, support.TESTFN) + for mode in ('w', 'r', 'r+', 'w+', 'a', 'a+'): + with self.subTest(mode), \ + codecs.open(support.TESTFN, mode, 'ascii') as file: + self.assertIsInstance(file, codecs.StreamReaderWriter) + + def test_undefined(self): + self.assertRaises(UnicodeError, codecs.encode, 'abc', 'undefined') + self.assertRaises(UnicodeError, codecs.decode, b'abc', 'undefined') + self.assertRaises(UnicodeError, codecs.encode, '', 'undefined') + self.assertRaises(UnicodeError, codecs.decode, b'', 'undefined') + for errors in ('strict', 'ignore', 'replace', 'backslashreplace'): + self.assertRaises(UnicodeError, + codecs.encode, 'abc', 'undefined', errors) + self.assertRaises(UnicodeError, + codecs.decode, b'abc', 'undefined', errors) + class StreamReaderTest(unittest.TestCase): def setUp(self): @@ -1801,13 +1831,10 @@ # "undefined" # The following encodings don't work in stateful mode -broken_unicode_with_streams = [ +broken_unicode_with_stateful = [ "punycode", "unicode_internal" ] -broken_incremental_coders = broken_unicode_with_streams + [ - "idna", -] class BasicUnicodeTest(unittest.TestCase, MixInCheckStateHandling): def test_basics(self): @@ -1827,7 +1854,7 @@ (chars, size) = codecs.getdecoder(encoding)(b) self.assertEqual(chars, s, "encoding=%r" % encoding) - if encoding not in broken_unicode_with_streams: + if encoding not in broken_unicode_with_stateful: # check stream reader/writer q = Queue(b"") writer = codecs.getwriter(encoding)(q) @@ -1845,7 +1872,7 @@ decodedresult += reader.read() self.assertEqual(decodedresult, s, "encoding=%r" % encoding) - if encoding not in broken_incremental_coders: + if encoding not in broken_unicode_with_stateful: # check incremental decoder/encoder and iterencode()/iterdecode() try: encoder = codecs.getincrementalencoder(encoding)() @@ -1894,7 +1921,7 @@ from _testcapi import codec_incrementalencoder, codec_incrementaldecoder s = "abc123" # all codecs should be able to encode these for encoding in all_unicode_encodings: - if encoding not in broken_incremental_coders: + if encoding not in broken_unicode_with_stateful: # check incremental decoder/encoder (fetched via the C API) try: cencoder = codec_incrementalencoder(encoding) @@ -1934,7 +1961,7 @@ for encoding in all_unicode_encodings: if encoding == "idna": # FIXME: See SF bug #1163178 continue - if encoding in broken_unicode_with_streams: + if encoding in broken_unicode_with_stateful: continue reader = codecs.getreader(encoding)(io.BytesIO(s.encode(encoding))) for t in range(5): @@ -1967,7 +1994,7 @@ # Check that getstate() and setstate() handle the state properly u = "abc123" for encoding in all_unicode_encodings: - if encoding not in broken_incremental_coders: + if encoding not in broken_unicode_with_stateful: self.check_state_handling_decode(encoding, u, u.encode(encoding)) self.check_state_handling_encode(encoding, u, u.encode(encoding)) @@ -2171,6 +2198,7 @@ f = io.BytesIO(b"\xc3\xbc") with codecs.EncodedFile(f, "latin-1", "utf-8") as ef: self.assertEqual(ef.read(), b"\xfc") + self.assertTrue(f.closed) def test_streamreaderwriter(self): f = io.BytesIO(b"\xc3\xbc") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -265,6 +265,10 @@ Tests ----- +- Issue #19548: Added some additional checks to test_codecs to ensure that + statements in the updated documentation remain accurate. Patch by Martin + Panter. + - Issue #22838: All test_re tests now work with unittest test discovery. - Issue #22173: Update lib2to3 tests to use unittest test discovery. @@ -297,6 +301,10 @@ Documentation ------------- +- Issue #19548: Update the codecs module documentation to better cover the + distinction between text encodings and other codecs, together with other + clarifications. Patch by Martin Panter. + - Issue #22914: Update the Python 2/3 porting HOWTO to describe a more automated approach. diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c --- a/Modules/_codecsmodule.c +++ b/Modules/_codecsmodule.c @@ -54,9 +54,9 @@ "register(search_function)\n\ \n\ Register a codec search function. Search functions are expected to take\n\ -one argument, the encoding name in all lower case letters, and return\n\ -a tuple of functions (encoder, decoder, stream_reader, stream_writer)\n\ -(or a CodecInfo object)."); +one argument, the encoding name in all lower case letters, and either\n\ +return None, or a tuple of functions (encoder, decoder, stream_reader,\n\ +stream_writer) (or a CodecInfo object)."); static PyObject *codec_register(PyObject *self, PyObject *search_function) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 17:27:16 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 06 Jan 2015 16:27:16 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fix_a_markup_e?= =?utf-8?q?rror_in_the_argparse_documentation=2E?= Message-ID: <20150106162648.72555.74400@psf.io> https://hg.python.org/cpython/rev/35b5ff543d4b changeset: 94055:35b5ff543d4b branch: 2.7 parent: 94049:eddcb6671a48 user: Berker Peksag date: Tue Jan 06 18:29:04 2015 +0200 summary: Fix a markup error in the argparse documentation. Reported by Jason Sachs on docs at . files: Doc/library/argparse.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1216,8 +1216,8 @@ which processes arguments from the command-line. Any object which follows this API may be passed as the ``action`` parameter to :meth:`add_argument`. -.. class:: Action(option_strings, dest, nargs=None, const=None, default=None, - type=None, choices=None, required=False, help=None, +.. class:: Action(option_strings, dest, nargs=None, const=None, default=None, \ + type=None, choices=None, required=False, help=None, \ metavar=None) Action objects are used by an ArgumentParser to represent the information needed -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 17:33:50 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 06 Jan 2015 16:33:50 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Silence_a_Sphi?= =?utf-8?q?nx_warning_in_ftplib=2Erst=2E?= Message-ID: <20150106163346.11571.53580@psf.io> https://hg.python.org/cpython/rev/67224c88144e changeset: 94056:67224c88144e branch: 2.7 user: Berker Peksag date: Tue Jan 06 18:36:02 2015 +0200 summary: Silence a Sphinx warning in ftplib.rst. ftplib.rst:61: WARNING: Block quote ends without a blank line; unexpect ed unindent. files: Doc/library/ftplib.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -57,7 +57,7 @@ .. class:: FTP_TLS([host[, user[, passwd[, acct[, keyfile[, certfile[, context[, timeout]]]]]]]]) - A :class:`FTP` subclass which adds TLS support to FTP as described in + A :class:`FTP` subclass which adds TLS support to FTP as described in :rfc:`4217`. Connect as usual to port 21 implicitly securing the FTP control connection before authenticating. Securing the data connection requires the user to -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 6 22:54:28 2015 From: python-checkins at python.org (georg.brandl) Date: Tue, 06 Jan 2015 21:54:28 +0000 Subject: [Python-checkins] =?utf-8?q?release=3A_comment_update?= Message-ID: <20150106215425.8751.16553@psf.io> https://hg.python.org/release/rev/d21e7f948b8b changeset: 92:d21e7f948b8b user: Georg Brandl date: Tue Jan 06 22:54:14 2015 +0100 summary: comment update files: add-to-pydotorg.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/add-to-pydotorg.py b/add-to-pydotorg.py --- a/add-to-pydotorg.py +++ b/add-to-pydotorg.py @@ -3,11 +3,11 @@ Script to add ReleaseFile objects for Python releases on the new pydotorg. To use (RELEASE is something like 3.3.5rc2): -* Copy this script to dinsdale (it needs access to all the release files). +* Copy this script to dl-files (it needs access to all the release files). You could also download all files, then you need to adapt the "ftp_root" string below. -* Make sure all download files are in place in the correct /data/ftp.python.org +* Make sure all download files are in place in the correct /srv/www.python.org subdirectory. * Create a new Release object via the Django admin (adding via API is -- Repository URL: https://hg.python.org/release From python-checkins at python.org Wed Jan 7 04:15:03 2015 From: python-checkins at python.org (nick.coghlan) Date: Wed, 07 Jan 2015 03:15:03 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319548=3A_clean_up?= =?utf-8?q?_merge_issues_in_codecs_docs?= Message-ID: <20150107031458.8767.41129@psf.io> https://hg.python.org/cpython/rev/20a5a56ce090 changeset: 94057:20a5a56ce090 parent: 94054:4d00d0109147 user: Nick Coghlan date: Wed Jan 07 13:14:47 2015 +1000 summary: Issue #19548: clean up merge issues in codecs docs Patch by Martin Panter to clean up some problems with the merge of the codecs docs changes from Python 3.4. files: Doc/library/codecs.rst | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -256,7 +256,6 @@ encodings. -.. _surrogateescape: .. _codec-base-classes: Codec Base Classes @@ -273,6 +272,7 @@ codec will handle encoding and decoding errors. +.. _surrogateescape: .. _error-handlers: Error Handlers @@ -319,7 +319,8 @@ | | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ | ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | -| | (only for encoding). | +| | (only for encoding). Implemented in | +| | :func:`namereplace_errors`. | +-------------------------+-----------------------------------------------+ | ``'surrogateescape'`` | On decoding, replace byte with individual | | | surrogate code ranging from ``U+DC80`` to | @@ -422,7 +423,8 @@ .. function:: namereplace_errors(exception) - Implements the ``namereplace`` error handling (for encoding only): the + Implements the ``'namereplace'`` error handling (for encoding with + :term:`text encodings ` only): the unencodable character is replaced by a ``\N{...}`` escape sequence. .. versionadded:: 3.5 -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 07:16:18 2015 From: python-checkins at python.org (raymond.hettinger) Date: Wed, 07 Jan 2015 06:16:18 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Minor_speed-up=2E__Use_loc?= =?utf-8?q?al_variable_instead_of_a_global_lookup=2E?= Message-ID: <20150107061616.11571.19695@psf.io> https://hg.python.org/cpython/rev/98376cf9133d changeset: 94058:98376cf9133d user: Raymond Hettinger date: Tue Jan 06 22:16:10 2015 -0800 summary: Minor speed-up. Use local variable instead of a global lookup. files: Lib/functools.py | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -98,7 +98,7 @@ 'Return a > b. Computed by @total_ordering from (not a < b) and (a != b).' op_result = self.__lt__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return not op_result and self != other def _le_from_lt(self, other): @@ -110,35 +110,35 @@ 'Return a >= b. Computed by @total_ordering from (not a < b).' op_result = self.__lt__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return not op_result def _ge_from_le(self, other): 'Return a >= b. Computed by @total_ordering from (not a <= b) or (a == b).' op_result = self.__le__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return not op_result or self == other def _lt_from_le(self, other): 'Return a < b. Computed by @total_ordering from (a <= b) and (a != b).' op_result = self.__le__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return op_result and self != other def _gt_from_le(self, other): 'Return a > b. Computed by @total_ordering from (not a <= b).' op_result = self.__le__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return not op_result def _lt_from_gt(self, other): 'Return a < b. Computed by @total_ordering from (not a > b) and (a != b).' op_result = self.__gt__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return not op_result and self != other def _ge_from_gt(self, other): @@ -150,28 +150,28 @@ 'Return a <= b. Computed by @total_ordering from (not a > b).' op_result = self.__gt__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return not op_result def _le_from_ge(self, other): 'Return a <= b. Computed by @total_ordering from (not a >= b) or (a == b).' op_result = self.__ge__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return not op_result or self == other def _gt_from_ge(self, other): 'Return a > b. Computed by @total_ordering from (a >= b) and (a != b).' op_result = self.__ge__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return op_result and self != other def _lt_from_ge(self, other): 'Return a < b. Computed by @total_ordering from (not a >= b).' op_result = self.__ge__(other) if op_result is NotImplemented: - return NotImplemented + return op_result return not op_result def total_ordering(cls): -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Wed Jan 7 09:39:40 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 07 Jan 2015 09:39:40 +0100 Subject: [Python-checkins] Daily reference leaks (4d00d0109147): sum=9 Message-ID: results for 4d00d0109147 on branch "default" -------------------------------------------- test_asyncio leaked [0, 3, 3] memory blocks, sum=6 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogDf86iG', '-x'] From python-checkins at python.org Wed Jan 7 18:14:57 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 17:14:57 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_expose_the_client=27s_ciph?= =?utf-8?q?er_suites_from_the_handshake_=28closes_=2323186=29?= Message-ID: <20150107171447.8767.17724@psf.io> https://hg.python.org/cpython/rev/bc34fcccaca3 changeset: 94059:bc34fcccaca3 user: Benjamin Peterson date: Wed Jan 07 11:14:26 2015 -0600 summary: expose the client's cipher suites from the handshake (closes #23186) files: Doc/library/ssl.rst | 12 +++++ Lib/ssl.py | 10 ++++ Lib/test/test_ssl.py | 17 +++++++ Misc/NEWS | 4 + Modules/_ssl.c | 72 ++++++++++++++++++++++--------- 5 files changed, 94 insertions(+), 21 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -925,6 +925,17 @@ version of the SSL protocol that defines its use, and the number of secret bits being used. If no connection has been established, returns ``None``. +.. method:: SSLSocket.shared_ciphers() + + Return the list of ciphers shared by the client during the handshake. Each + entry of the returned list is a three-value tuple containing the name of the + cipher, the version of the SSL protocol that defines its use, and the number + of secret bits the cipher uses. :meth:`~SSLSocket.shared_ciphers` returns + ``None`` if no connection has been established or the socket is a client + socket. + + .. versionadded:: 3.5 + .. method:: SSLSocket.compression() Return the compression algorithm being used as a string, or ``None`` @@ -1784,6 +1795,7 @@ - :meth:`~SSLSocket.getpeercert` - :meth:`~SSLSocket.selected_npn_protocol` - :meth:`~SSLSocket.cipher` + - :meth:`~SSLSocket.shared_ciphers` - :meth:`~SSLSocket.compression` - :meth:`~SSLSocket.pending` - :meth:`~SSLSocket.do_handshake` diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -572,6 +572,10 @@ ssl_version, secret_bits)``.""" return self._sslobj.cipher() + def shared_ciphers(self): + """Return the ciphers shared by the client during the handshake.""" + return self._sslobj.shared_ciphers() + def compression(self): """Return the current compression algorithm in use, or ``None`` if compression was not negotiated or not supported by one of the peers.""" @@ -784,6 +788,12 @@ else: return self._sslobj.cipher() + def shared_ciphers(self): + self._checkClosed() + if not self._sslobj: + return None + return self._sslobj.shared_ciphers() + def compression(self): self._checkClosed() if not self._sslobj: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1698,11 +1698,13 @@ sslobj = ctx.wrap_bio(incoming, outgoing, False, 'svn.python.org') self.assertIs(sslobj._sslobj.owner, sslobj) self.assertIsNone(sslobj.cipher()) + self.assertIsNone(sslobj.shared_ciphers()) self.assertRaises(ValueError, sslobj.getpeercert) if 'tls-unique' in ssl.CHANNEL_BINDING_TYPES: self.assertIsNone(sslobj.get_channel_binding('tls-unique')) self.ssl_io_loop(sock, incoming, outgoing, sslobj.do_handshake) self.assertTrue(sslobj.cipher()) + self.assertIsNone(sslobj.shared_ciphers()) self.assertTrue(sslobj.getpeercert()) if 'tls-unique' in ssl.CHANNEL_BINDING_TYPES: self.assertTrue(sslobj.get_channel_binding('tls-unique')) @@ -1776,6 +1778,7 @@ self.close() return False else: + self.server.shared_ciphers.append(self.sslconn.shared_ciphers()) if self.server.context.verify_mode == ssl.CERT_REQUIRED: cert = self.sslconn.getpeercert() if support.verbose and self.server.chatty: @@ -1891,6 +1894,7 @@ self.flag = None self.active = False self.selected_protocols = [] + self.shared_ciphers = [] self.conn_errors = [] threading.Thread.__init__(self) self.daemon = True @@ -2121,6 +2125,7 @@ }) s.close() stats['server_npn_protocols'] = server.selected_protocols + stats['server_shared_ciphers'] = server.shared_ciphers return stats def try_protocol_combo(server_protocol, client_protocol, expect_success, @@ -3157,6 +3162,18 @@ self.assertEqual(cm.exception.reason, 'TLSV1_ALERT_INTERNAL_ERROR') self.assertIn("TypeError", stderr.getvalue()) + def test_shared_ciphers(self): + server_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + client_context.set_ciphers("3DES") + server_context.set_ciphers("3DES:AES") + stats = server_params_test(client_context, server_context) + ciphers = stats['server_shared_ciphers'][0] + self.assertGreater(len(ciphers), 0) + for name, tls_version, bits in ciphers: + self.assertIn("DES-CBC3-", name) + self.assertEqual(bits, 112) + def test_read_write_after_close_raises_valuerror(self): context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) context.verify_mode = ssl.CERT_REQUIRED diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -199,6 +199,10 @@ Library ------- +- Issue #23186: Add ssl.SSLObject.shared_ciphers() and + ssl.SSLSocket.shared_ciphers() to fetch the client's list ciphers sent at + handshake. + - Issue #23143: Remove compatibility with OpenSSLs older than 0.9.8. - Issue #23132: Improve performance and introspection support of comparison diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -1360,54 +1360,83 @@ peer certificate, or None if no certificate was provided. This will\n\ return the certificate even if it wasn't validated."); -static PyObject *PySSL_cipher (PySSLSocket *self) { - - PyObject *retval, *v; - const SSL_CIPHER *current; - char *cipher_name; - char *cipher_protocol; - - if (self->ssl == NULL) - Py_RETURN_NONE; - current = SSL_get_current_cipher(self->ssl); - if (current == NULL) - Py_RETURN_NONE; - - retval = PyTuple_New(3); +static PyObject * +cipher_to_tuple(const SSL_CIPHER *cipher) +{ + const char *cipher_name, *cipher_protocol; + PyObject *v, *retval = PyTuple_New(3); if (retval == NULL) return NULL; - cipher_name = (char *) SSL_CIPHER_get_name(current); + cipher_name = SSL_CIPHER_get_name(cipher); if (cipher_name == NULL) { Py_INCREF(Py_None); PyTuple_SET_ITEM(retval, 0, Py_None); } else { v = PyUnicode_FromString(cipher_name); if (v == NULL) - goto fail0; + goto fail; PyTuple_SET_ITEM(retval, 0, v); } - cipher_protocol = (char *) SSL_CIPHER_get_version(current); + + cipher_protocol = SSL_CIPHER_get_version(cipher); if (cipher_protocol == NULL) { Py_INCREF(Py_None); PyTuple_SET_ITEM(retval, 1, Py_None); } else { v = PyUnicode_FromString(cipher_protocol); if (v == NULL) - goto fail0; + goto fail; PyTuple_SET_ITEM(retval, 1, v); } - v = PyLong_FromLong(SSL_CIPHER_get_bits(current, NULL)); + + v = PyLong_FromLong(SSL_CIPHER_get_bits(cipher, NULL)); if (v == NULL) - goto fail0; + goto fail; PyTuple_SET_ITEM(retval, 2, v); + return retval; - fail0: + fail: Py_DECREF(retval); return NULL; } +static PyObject *PySSL_shared_ciphers(PySSLSocket *self) +{ + STACK_OF(SSL_CIPHER) *ciphers; + int i; + PyObject *res; + + if (!self->ssl->session || !self->ssl->session->ciphers) + Py_RETURN_NONE; + ciphers = self->ssl->session->ciphers; + res = PyList_New(sk_SSL_CIPHER_num(ciphers)); + if (!res) + return NULL; + for (i = 0; i < sk_SSL_CIPHER_num(ciphers); i++) { + PyObject *tup = cipher_to_tuple(sk_SSL_CIPHER_value(ciphers, i)); + if (!tup) { + Py_DECREF(res); + return NULL; + } + PyList_SET_ITEM(res, i, tup); + } + return res; +} + +static PyObject *PySSL_cipher (PySSLSocket *self) +{ + const SSL_CIPHER *current; + + if (self->ssl == NULL) + Py_RETURN_NONE; + current = SSL_get_current_cipher(self->ssl); + if (current == NULL) + Py_RETURN_NONE; + return cipher_to_tuple(current); +} + static PyObject *PySSL_version(PySSLSocket *self) { const char *version; @@ -2019,6 +2048,7 @@ {"peer_certificate", (PyCFunction)PySSL_peercert, METH_VARARGS, PySSL_peercert_doc}, {"cipher", (PyCFunction)PySSL_cipher, METH_NOARGS}, + {"shared_ciphers", (PyCFunction)PySSL_shared_ciphers, METH_NOARGS}, {"version", (PyCFunction)PySSL_version, METH_NOARGS}, #ifdef OPENSSL_NPN_NEGOTIATED {"selected_npn_protocol", (PyCFunction)PySSL_selected_npn_protocol, METH_NOARGS}, -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 18:32:25 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 17:32:25 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_explain_None_can_be_return?= =?utf-8?q?ed?= Message-ID: <20150107173205.22419.96082@psf.io> https://hg.python.org/cpython/rev/7c208844f560 changeset: 94060:7c208844f560 user: Benjamin Peterson date: Wed Jan 07 11:26:50 2015 -0600 summary: explain None can be returned files: Lib/ssl.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -573,7 +573,9 @@ return self._sslobj.cipher() def shared_ciphers(self): - """Return the ciphers shared by the client during the handshake.""" + """Return the a list of ciphers shared by the client during the + handshake or None if this is not a valid server connection. + """ return self._sslobj.shared_ciphers() def compression(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 18:32:25 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 17:32:25 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_use_SSL=5Fget=5Fsession?= Message-ID: <20150107173205.72581.21726@psf.io> https://hg.python.org/cpython/rev/dd41bc9c5f60 changeset: 94061:dd41bc9c5f60 user: Benjamin Peterson date: Wed Jan 07 11:32:00 2015 -0600 summary: use SSL_get_session files: Modules/_ssl.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -1404,13 +1404,14 @@ static PyObject *PySSL_shared_ciphers(PySSLSocket *self) { + SSL_SESSION *sess = SSL_get_session(self->ssl); STACK_OF(SSL_CIPHER) *ciphers; int i; PyObject *res; - if (!self->ssl->session || !self->ssl->session->ciphers) + if (!sess || !sess->ciphers) Py_RETURN_NONE; - ciphers = self->ssl->session->ciphers; + ciphers = sess->ciphers; res = PyList_New(sk_SSL_CIPHER_num(ciphers)); if (!res) return NULL; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 18:35:27 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 17:35:27 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_remove_apparently_wrong_as?= =?utf-8?q?sertion_about_des_bit_size?= Message-ID: <20150107173355.72575.75267@psf.io> https://hg.python.org/cpython/rev/9c68831ff153 changeset: 94062:9c68831ff153 user: Benjamin Peterson date: Wed Jan 07 11:33:51 2015 -0600 summary: remove apparently wrong assertion about des bit size files: Lib/test/test_ssl.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3172,7 +3172,6 @@ self.assertGreater(len(ciphers), 0) for name, tls_version, bits in ciphers: self.assertIn("DES-CBC3-", name) - self.assertEqual(bits, 112) def test_read_write_after_close_raises_valuerror(self): context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 18:42:54 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 17:42:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_force_test_server_to_speak?= =?utf-8?q?_tlsv1?= Message-ID: <20150107174243.22417.97506@psf.io> https://hg.python.org/cpython/rev/d2fbefb1c818 changeset: 94063:d2fbefb1c818 user: Benjamin Peterson date: Wed Jan 07 11:42:38 2015 -0600 summary: force test server to speak tlsv1 files: Lib/test/test_ssl.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3163,7 +3163,7 @@ self.assertIn("TypeError", stderr.getvalue()) def test_shared_ciphers(self): - server_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) client_context.set_ciphers("3DES") server_context.set_ciphers("3DES:AES") -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 19:59:36 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 18:59:36 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_include_some_more_ciphers?= Message-ID: <20150107185926.72551.12533@psf.io> https://hg.python.org/cpython/rev/15ce5fee0508 changeset: 94064:15ce5fee0508 user: Benjamin Peterson date: Wed Jan 07 12:59:20 2015 -0600 summary: include some more ciphers files: Lib/test/test_ssl.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3165,13 +3165,13 @@ def test_shared_ciphers(self): server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - client_context.set_ciphers("3DES") - server_context.set_ciphers("3DES:AES") + client_context.set_ciphers("3DES:DES") + server_context.set_ciphers("3DES:DES:AES") stats = server_params_test(client_context, server_context) ciphers = stats['server_shared_ciphers'][0] self.assertGreater(len(ciphers), 0) for name, tls_version, bits in ciphers: - self.assertIn("DES-CBC3-", name) + self.assertIn("DES", name.split("-")) def test_read_write_after_close_raises_valuerror(self): context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 20:15:56 2015 From: python-checkins at python.org (berker.peksag) Date: Wed, 07 Jan 2015 19:15:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwNDg3?= =?utf-8?q?=3A_Clarify_meaning_of_=22side_effect=22_in_the_magic_mock_docu?= =?utf-8?q?mentation=2E?= Message-ID: <20150107191544.125888.92064@psf.io> https://hg.python.org/cpython/rev/230a1bfb0f59 changeset: 94065:230a1bfb0f59 branch: 3.4 parent: 94053:0646eee8296a user: Berker Peksag date: Wed Jan 07 21:15:02 2015 +0200 summary: Issue #20487: Clarify meaning of "side effect" in the magic mock documentation. Patch by A.M. Kuchling. files: Doc/library/unittest.mock.rst | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -1678,9 +1678,10 @@ >>> object() in mock False -The two equality method, :meth:`__eq__` and :meth:`__ne__`, are special. -They do the default equality comparison on identity, using a side -effect, unless you change their return value to return something else: +The two equality methods, :meth:`__eq__` and :meth:`__ne__`, are special. +They do the default equality comparison on identity, using the +:attr:`~Mock.side_effect` attribute, unless you change their return value to +return something else:: >>> MagicMock() == 3 False -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 20:15:56 2015 From: python-checkins at python.org (berker.peksag) Date: Wed, 07 Jan 2015 19:15:56 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320487=3A_Clarify_meaning_of_=22side_effect=22_i?= =?utf-8?q?n_the_magic_mock_documentation=2E?= Message-ID: <20150107191545.72573.65161@psf.io> https://hg.python.org/cpython/rev/3cf91d2aeab3 changeset: 94066:3cf91d2aeab3 parent: 94064:15ce5fee0508 parent: 94065:230a1bfb0f59 user: Berker Peksag date: Wed Jan 07 21:15:33 2015 +0200 summary: Issue #20487: Clarify meaning of "side effect" in the magic mock documentation. Patch by A.M. Kuchling. files: Doc/library/unittest.mock.rst | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -1720,9 +1720,10 @@ >>> object() in mock False -The two equality method, :meth:`__eq__` and :meth:`__ne__`, are special. -They do the default equality comparison on identity, using a side -effect, unless you change their return value to return something else: +The two equality methods, :meth:`__eq__` and :meth:`__ne__`, are special. +They do the default equality comparison on identity, using the +:attr:`~Mock.side_effect` attribute, unless you change their return value to +return something else:: >>> MagicMock() == 3 False -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 20:29:27 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 19:29:27 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_everyone_should_support_AE?= =?utf-8?q?S_ciphers?= Message-ID: <20150107192924.72559.61214@psf.io> https://hg.python.org/cpython/rev/7a9a2a9c2e2b changeset: 94067:7a9a2a9c2e2b user: Benjamin Peterson date: Wed Jan 07 13:28:40 2015 -0600 summary: everyone should support AES ciphers files: Lib/test/test_ssl.py | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3165,13 +3165,14 @@ def test_shared_ciphers(self): server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - client_context.set_ciphers("3DES:DES") - server_context.set_ciphers("3DES:DES:AES") + client_context.set_ciphers("AES128") + server_context.set_ciphers("AES128:AES256") stats = server_params_test(client_context, server_context) ciphers = stats['server_shared_ciphers'][0] self.assertGreater(len(ciphers), 0) for name, tls_version, bits in ciphers: - self.assertIn("DES", name.split("-")) + self.assertIn("AES128", name.split("-")) + self.assertEqual(bits, 128) def test_read_write_after_close_raises_valuerror(self): context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 21:21:32 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 20:21:32 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_rc4_is_a_long_time_favorit?= =?utf-8?q?e?= Message-ID: <20150107202128.8745.8287@psf.io> https://hg.python.org/cpython/rev/803898b6bee3 changeset: 94068:803898b6bee3 user: Benjamin Peterson date: Wed Jan 07 14:21:22 2015 -0600 summary: rc4 is a long time favorite files: Lib/test/test_ssl.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3165,8 +3165,8 @@ def test_shared_ciphers(self): server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - client_context.set_ciphers("AES128") - server_context.set_ciphers("AES128:AES256") + client_context.set_ciphers("RC4") + server_context.set_ciphers("AES:RC4") stats = server_params_test(client_context, server_context) ciphers = stats['server_shared_ciphers'][0] self.assertGreater(len(ciphers), 0) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 7 21:30:00 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 07 Jan 2015 20:30:00 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_fix_assertions_after_ciphe?= =?utf-8?q?rs_were_changed?= Message-ID: <20150107202955.11561.11225@psf.io> https://hg.python.org/cpython/rev/a969b9c53675 changeset: 94069:a969b9c53675 user: Benjamin Peterson date: Wed Jan 07 14:29:45 2015 -0600 summary: fix assertions after ciphers were changed files: Lib/test/test_ssl.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3171,8 +3171,7 @@ ciphers = stats['server_shared_ciphers'][0] self.assertGreater(len(ciphers), 0) for name, tls_version, bits in ciphers: - self.assertIn("AES128", name.split("-")) - self.assertEqual(bits, 128) + self.assertIn("RC4", name.split("-")) def test_read_write_after_close_raises_valuerror(self): context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 03:03:44 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 08 Jan 2015 02:03:44 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_try_using_AES256?= Message-ID: <20150108020331.22419.65654@psf.io> https://hg.python.org/cpython/rev/e42301eb3ce8 changeset: 94070:e42301eb3ce8 user: Benjamin Peterson date: Wed Jan 07 20:03:27 2015 -0600 summary: try using AES256 files: Lib/test/test_ssl.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3165,13 +3165,13 @@ def test_shared_ciphers(self): server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - client_context.set_ciphers("RC4") - server_context.set_ciphers("AES:RC4") + client_context.set_ciphers("AES256") + server_context.set_ciphers("RC4:AES") stats = server_params_test(client_context, server_context) ciphers = stats['server_shared_ciphers'][0] self.assertGreater(len(ciphers), 0) for name, tls_version, bits in ciphers: - self.assertIn("RC4", name.split("-")) + self.assertIn("AES256", name.split("-")) def test_read_write_after_close_raises_valuerror(self): context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 03:31:04 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 08 Jan 2015 02:31:04 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_drop_256?= Message-ID: <20150108023102.22411.32872@psf.io> https://hg.python.org/cpython/rev/c65c054a605d changeset: 94071:c65c054a605d user: Benjamin Peterson date: Wed Jan 07 20:30:59 2015 -0600 summary: drop 256 files: Lib/test/test_ssl.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3165,13 +3165,13 @@ def test_shared_ciphers(self): server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - client_context.set_ciphers("AES256") + client_context.set_ciphers("AES") server_context.set_ciphers("RC4:AES") stats = server_params_test(client_context, server_context) ciphers = stats['server_shared_ciphers'][0] self.assertGreater(len(ciphers), 0) for name, tls_version, bits in ciphers: - self.assertIn("AES256", name.split("-")) + self.assertIn("AES", name) def test_read_write_after_close_raises_valuerror(self): context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 03:52:46 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 08 Jan 2015 02:52:46 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_reorder_cipher_prefs?= Message-ID: <20150108025243.11591.95302@psf.io> https://hg.python.org/cpython/rev/9968783893e5 changeset: 94072:9968783893e5 user: Benjamin Peterson date: Wed Jan 07 20:52:40 2015 -0600 summary: reorder cipher prefs files: Lib/test/test_ssl.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3166,7 +3166,7 @@ server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) client_context.set_ciphers("AES") - server_context.set_ciphers("RC4:AES") + server_context.set_ciphers("AES:RC4") stats = server_params_test(client_context, server_context) ciphers = stats['server_shared_ciphers'][0] self.assertGreater(len(ciphers), 0) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 04:06:12 2015 From: python-checkins at python.org (nick.coghlan) Date: Thu, 08 Jan 2015 03:06:12 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_474=3A_Updated_forge=2Epy?= =?utf-8?q?thon=2Eorg_proposal?= Message-ID: <20150108030602.125880.91336@psf.io> https://hg.python.org/peps/rev/dadbf78cddcb changeset: 5657:dadbf78cddcb user: Nick Coghlan date: Thu Jan 08 13:05:54 2015 +1000 summary: PEP 474: Updated forge.python.org proposal files: pep-0474.txt | 242 +++++++++++++++++++++++++++++++------- 1 files changed, 197 insertions(+), 45 deletions(-) diff --git a/pep-0474.txt b/pep-0474.txt --- a/pep-0474.txt +++ b/pep-0474.txt @@ -23,18 +23,11 @@ for CPython itself (see PEP 462 in relation to that). -PEP Deferral -============ - -This PEP is deferred largely because I don't currently have time to work on -it. If anyone would like to take it over, let me know. - - Proposal ======== -This PEP proposes that, once the Kallithea project has made an official -release, that a Kallithea instance be deployed as "forge.python.org". +This PEP proposes that an instance of the self-hosted Kallithea code +repository management system be deployed as "forge.python.org". Individual repositories (such as the developer guide or the PEPs repository) may then be migrated from the existing hg.python.org infrastructure to the @@ -42,9 +35,24 @@ will need to decide whether to retain a read-only mirror on hg.python.org, or whether to just migrate wholesale to the new location. -This would not be a general purpose hosting site for arbitrary Python -projects, but a more narrowly focused site akin to the existing -hg.python.org. +In addition to supporting read-only mirrors on hg.python.org, +forge.python.org will also aim to support hosting mirrors on popular +proprietary hosting sites like GitHub and BitBucket. The aim will be to +allow users familiar with these sites to submit and discuss pull requests +using their preferred workflow, with forge.python.org automatically bringing +those contributions over to the master repository. + +Given the availability and popularity of commercially backed "free for open +source projects" repository hosting services, this would not be a general +purpose hosting site for arbitrary Python projects. The initial focus will be +specifically on CPython and other repositories currently hosted on +hg.python.org. In the future, this could potentially be expanded to +consolidating other PSF managed repositories that are currently externally +hosted to gain access to a pull request based workflow, such as the +repository for the python.org Django application. As with the initial +migrations, any such future migrations would be considered on a case-by-case +basis, taking into account the preferences of the primary users of each +repository. Rationale @@ -64,36 +72,42 @@ The key requirements proposed for a PSF provided software forge are: -* Must support self-hosting on PSF infrastructure without ongoing fees -* Must support Mercurial (for consistency with existing tooling) -* Must support simple "pull request" style workflows -* Must support online editing for simple changes +* MUST support simple "pull request" style workflows +* MUST support online editing for simple changes +* MUST be backed by an active development organisation (community or + commercial) -Ideally, the chosen solution would also be a fully open source application -written in Python. +Additional recommended requirements that are satisfied by this proposal, +but may be negotiable if a sufficiently compelling alternative is presented: -The requirement for self-hosting without ongoing fees rules out the +* SHOULD support self-hosting on PSF infrastructure without ongoing fees +* SHOULD be a fully open source application written in Python +* SHOULD support Mercurial (for consistency with existing tooling) +* SHOULD support Git (to provide that option to users that prefer it) +* SHOULD allow users of git and Mercurial clients to transparently + collaborate on the same repository +* SHOULD be open to customisation to meet the needs of CPython core + development, including providing a potential path forward for the + proposed migration to a core reviewer model in PEP 462 + + +The preference for self-hosting without ongoing fees rules out the free-as-in-beer providers like GitHub and BitBucket, in addition to the various proprietary source code management offerings. -The requirement for Mercurial support not only rules out GitHub, but also +The preference for Mercurial support not only rules out GitHub, but also other Git-only solutions like GitLab and Gitorious. -The requirement for online editing support rules out the Apache +The hard requirement for online editing support rules out the Apache Allura/HgForge combination. -That leaves two main contenders from a technical perspective: +The preference for a fully open source solution rules out RhodeCode. -* `RhodeCode `__ -* `Kallithea SCM `__ +Of the various options considered by the author of this proposal, that +leaves `Kallithea SCM `__ as the proposed +foundation for a forge.python.org service. -The `legal uncertainty -`__ over the -interaction between RhodeCode's current "Business Source" licensing model and -the various GPL components it relies on currently make it unclear whether it -is legally permissible to deploy it. - -By contrast, Kallithea is a full GPLv3 application (derived from the clearly +Kallithea is a full GPLv3 application (derived from the clearly and unambiguously GPLv3 licensed components of RhodeCode) that is being developed under the auspices of the `Software Freedom Conservancy `__. The @@ -106,7 +120,7 @@ Twisted, Gevent, BuildBot and PyPy. -Perceived Benefits +Intended Benefits ================== The primary benefit of deploying Kallithea as forge.python.org is that @@ -122,24 +136,97 @@ installation. -Technical Challenges -==================== +Sustaining Engineering Considerations +===================================== + +Even with its current workflow, CPython itself remains one of the largest +open source projects in the world (in the +`top 2% ` +of projects tracked on OpenHub). Unfortunately, we have been significantly +less effective at encouraging contributions to the projects that make up +CPython's workflow infrastructure, including ensuring that our installations +track upstream, and that wherever feasible, our own customisations are +contributed back to the original project. + +As such, a core component of this proposal is to actively engage with the +upstream Kallithea community to lower the barriers to working with and on +the Kallithea SCM, as well as with the PSF Infrastructure team to ensure +the forge.python.org service integrates cleanly with the PSF's infrastructure +automation. + +This approach aims to provide a number of key benefits: + +* allowing those of us contributing to maintenance of this service to be + as productive as possible in the time we have available +* offering a compelling professional development opportunity to those + volunteers that choose to participate in maintenance of this service +* making the Kallithea project itself more attractive to other potential + users by making it as easy as possible to adopt, deploy and manage +* as a result of the above benefits, attracting sufficient contributors both + in the upstream Kallithea community, and within the CPython infrastructure + community, to allow the forge.python.org service to evolve effectively to + meet changing developer expectations + +Some initial steps have already been taken to address these sustaining +engineering concerns: + +* Tymoteusz Jankowski has been working with Donald Stufft to work out `what + would be involved `__ + in deploying Kallithea using the PSF's Salt based infrastructure automation. +* Graham Dumpleton and I have been working on + `making it easy + +`__ to deploy + demonstration Kallithea instances to the free tier of Red Hat's open source + hosting service, OpenShift Online. (See the comments on that post, or the + `quickstart issue tracker + `__ for links to + Graham's follow on work) + +The next major step to be undertaken is to come up with a local development +workflow that allows contributors on Windows, Mac OS X and Linux to run +the Kallithea tests locally, without interfering with the operation of +their own system. The currently planned approach for this is to focus on +Vagrant, which is a popular automated virtual machine management system +specifically aimed at developers running local VMs for testing purposes. +The `Vagrant based development guidelines +`__ +for +OpenShift Origin provide an extended example of the kind of workflow this +approach enables. It's also worth noting that Vagrant is one of the options +for working with a local build of the `main python.org website +`__. + +If these workflow proposals end up working well for Kallithea, they may also +be worth proposing for use by the upstream projects backing other PSF and +CPython infrastructure services, including Roundup, BuildBot, and the main +python.org web site. + + +Technical Concerns and Challenges +================================= + +Introducing a new service into the CPython infrastructure presents a number +of interesting technical concerns and challenges. This section covers several +of the most significant ones. User account management ----------------------- -Ideally we'd be able to offer a single account that spans all python.org -services, including Kallithea, Roundup/Rietveld, PyPI and the back end for -the new python.org site, but actually implementing that would be a distinct -infrastructure project, independent of this PEP. +Ideally we'd like to be able to offer a single account that spans all +python.org services, including Kallithea, Roundup/Rietveld, PyPI and the +back end for the new python.org site, but actually implementing that would +be a distinct infrastructure project, independent of this PEP. -A potentially simpler near term solution would be if Kallithea's user -account management could be integrated with the existing account management -in Roundup, similar to what was done for Rietveld. However, if that also -turns out to be impractical in the near term, we would merely end up with -another identity silo to be integrated at a later date, suggesting that -this doesn't need to be considered a blocker for an initial Kallithea -deployment. +For the initial rollout of forge.python.org, we will likely create yet another +identity silo within the PSF infrastructure. A potentially superior +alternative would be to add support for `python-social-auth +`__ to Kallithea, but actually +doing so would not be a requirement for the initial rollout of the service +(the main technical concern there is that Kallithea is a Pylons application +that has not yet been ported to Pyramid, so integration will require either +adding a Pylons backend to python-social-auth, or else embarking on the +Pyramid migration in Kallithea). Breaking existing SSH access and links for Mercurial repositories @@ -153,6 +240,71 @@ break). +Integration with Roundup +------------------------ + +Kallithea provides configurable issue tracker integration. This will need +to be set up appropriately to integrate with the Roundup issue tracker at +bugs.python.org before the initial rollout of the forge.python.org service. + + +Accepting pull requests on GitHub and BitBucket +----------------------------------------------- + +The initial rollout of forge.python.org would support publication of read-only +mirrors, both on hg.python.org and other services, as that is a relatively +straightforward operation that can be implemented in a commit hook. + +While a highly desirable feature, accepting pull requests on external +services, and mirroring them as submissions to the master repositories on +forge.python.org is a more complex problem, and would likely not be included +as part of the initial rollout of the forge.python.org service. + + +Transparent Git and Mercurial interoperability +---------------------------------------------- + +Kallithea's native support for both Git and Mercurial offers an opportunity +to make it relatively straightforward for developers to use the client +of their choice to interact with repositories hosted on forge.python.org. + +This transparent interoperability does *not* exist yet, but running our own +multi-VCS repository hosting service provides the opportunity to make this +capability a reality, rather than passively waiting for a proprietary +provider to deign to provide a feature that likely isn't in their commercial +interest. There's a significant misalignment of incentives between open +source communities and commercial providers in this particular area, as even +though offering VCS client choice can significantly reduce community friction +by eliminating the need for projects to make autocratic decisions that force +particular tooling choices on potential contributors, top down enforcement +of tool selection (regardless of developer preference) is currently still +the norm in the corporate and other organisational environments that produce +GitHub and Atlassian's paying customers. + +Prior to acceptance, in the absence of transparent interoperability, this PEP +should propose specific recommendations for inclusion in the CPython +developer's guide for using git-hg to create pull requests against +forge.python.org hosted Mercurial repositories. + + +Future Implications for CPython Core Development +================================================ + +The workflow requirements for the main CPython development repository are +significantly more complex than those for the repositories being discussed +in this PEP. These concerns are covered in more detail in PEP 462. + +Given Guido's recommendation to replace Reitveld with a more actively +maintained code review system, my current plan is to rewrite that PEP to +use Kallithea as the proposed glue layer, with enhanced Kallithea pull +requests eventually replacing the current practice of uploading patche files +directly to the issue tracker. + +I've also started working with Pierre Yves-David on a `custom Mercurial +extension `__ +that automates some aspects of the CPython core development workflow. + + Copyright ========= -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Thu Jan 8 04:08:11 2015 From: python-checkins at python.org (nick.coghlan) Date: Thu, 08 Jan 2015 03:08:11 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_I_never_get_Rietveld_right_fi?= =?utf-8?b?cnN0IHRpbWUuLi4=?= Message-ID: <20150108030759.8759.71976@psf.io> https://hg.python.org/peps/rev/b4e78f3d80cd changeset: 5658:b4e78f3d80cd user: Nick Coghlan date: Thu Jan 08 13:07:52 2015 +1000 summary: I never get Rietveld right first time... files: pep-0474.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0474.txt b/pep-0474.txt --- a/pep-0474.txt +++ b/pep-0474.txt @@ -294,7 +294,7 @@ significantly more complex than those for the repositories being discussed in this PEP. These concerns are covered in more detail in PEP 462. -Given Guido's recommendation to replace Reitveld with a more actively +Given Guido's recommendation to replace Rietveld with a more actively maintained code review system, my current plan is to rewrite that PEP to use Kallithea as the proposed glue layer, with enhanced Kallithea pull requests eventually replacing the current practice of uploading patche files -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Thu Jan 8 04:24:53 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 08 Jan 2015 03:24:53 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_trying_again?= Message-ID: <20150108032446.125894.99202@psf.io> https://hg.python.org/cpython/rev/4ee1a459540a changeset: 94073:4ee1a459540a user: Benjamin Peterson date: Wed Jan 07 21:21:34 2015 -0600 summary: trying again files: Lib/test/test_ssl.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3165,13 +3165,13 @@ def test_shared_ciphers(self): server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - client_context.set_ciphers("AES") + client_context.set_ciphers("RC4") server_context.set_ciphers("AES:RC4") stats = server_params_test(client_context, server_context) ciphers = stats['server_shared_ciphers'][0] self.assertGreater(len(ciphers), 0) for name, tls_version, bits in ciphers: - self.assertIn("AES", name) + self.assertIn("RC4", name.split("-")) def test_read_write_after_close_raises_valuerror(self): context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 05:12:47 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 08 Jan 2015 04:12:47 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_enable_cert_validation_in_?= =?utf-8?q?test?= Message-ID: <20150108041246.8745.96180@psf.io> https://hg.python.org/cpython/rev/b89c84a4426a changeset: 94074:b89c84a4426a user: Benjamin Peterson date: Wed Jan 07 22:12:43 2015 -0600 summary: enable cert validation in test files: Lib/test/test_ssl.py | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3164,7 +3164,10 @@ def test_shared_ciphers(self): server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) - client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + server_context.load_cert_chain(SIGNED_CERTFILE) + client_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + client_context.verify_mode = ssl.CERT_REQUIRED + client_context.load_verify_locations(SIGNING_CA) client_context.set_ciphers("RC4") server_context.set_ciphers("AES:RC4") stats = server_params_test(client_context, server_context) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 05:49:35 2015 From: python-checkins at python.org (terry.reedy) Date: Thu, 08 Jan 2015 04:49:35 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <20150108044934.11589.19426@psf.io> https://hg.python.org/cpython/rev/58674a027d10 changeset: 94077:58674a027d10 parent: 94074:b89c84a4426a parent: 94076:364e44a49a31 user: Terry Jan Reedy date: Wed Jan 07 23:49:06 2015 -0500 summary: Merge with 3.4 files: Lib/idlelib/testcode.py | 31 ----------------------------- 1 files changed, 0 insertions(+), 31 deletions(-) diff --git a/Lib/idlelib/testcode.py b/Lib/idlelib/testcode.py deleted file mode 100644 --- a/Lib/idlelib/testcode.py +++ /dev/null @@ -1,31 +0,0 @@ -import string - -def f(): - a = 0 - b = 1 - c = 2 - d = 3 - e = 4 - g() - -def g(): - h() - -def h(): - i() - -def i(): - j() - -def j(): - k() - -def k(): - l() - -l = lambda: test() - -def test(): - string.capwords(1) - -f() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 05:49:35 2015 From: python-checkins at python.org (terry.reedy) Date: Thu, 08 Jan 2015 04:49:35 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMTg0?= =?utf-8?q?=3A_delete_unused_idlelib_file=2E?= Message-ID: <20150108044934.11569.3337@psf.io> https://hg.python.org/cpython/rev/1c3f8d044589 changeset: 94075:1c3f8d044589 branch: 2.7 parent: 94056:67224c88144e user: Terry Jan Reedy date: Wed Jan 07 23:48:28 2015 -0500 summary: Issue #23184: delete unused idlelib file. files: Lib/idlelib/testcode.py | 31 ----------------------------- 1 files changed, 0 insertions(+), 31 deletions(-) diff --git a/Lib/idlelib/testcode.py b/Lib/idlelib/testcode.py deleted file mode 100644 --- a/Lib/idlelib/testcode.py +++ /dev/null @@ -1,31 +0,0 @@ -import string - -def f(): - a = 0 - b = 1 - c = 2 - d = 3 - e = 4 - g() - -def g(): - h() - -def h(): - i() - -def i(): - j() - -def j(): - k() - -def k(): - l() - -l = lambda: test() - -def test(): - string.capwords(1) - -f() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 05:49:35 2015 From: python-checkins at python.org (terry.reedy) Date: Thu, 08 Jan 2015 04:49:35 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTg0?= =?utf-8?q?=3A_delete_unused_idlelib_file=2E?= Message-ID: <20150108044934.72555.53433@psf.io> https://hg.python.org/cpython/rev/364e44a49a31 changeset: 94076:364e44a49a31 branch: 3.4 parent: 94065:230a1bfb0f59 user: Terry Jan Reedy date: Wed Jan 07 23:48:46 2015 -0500 summary: Issue #23184: delete unused idlelib file. files: Lib/idlelib/testcode.py | 31 ----------------------------- 1 files changed, 0 insertions(+), 31 deletions(-) diff --git a/Lib/idlelib/testcode.py b/Lib/idlelib/testcode.py deleted file mode 100644 --- a/Lib/idlelib/testcode.py +++ /dev/null @@ -1,31 +0,0 @@ -import string - -def f(): - a = 0 - b = 1 - c = 2 - d = 3 - e = 4 - g() - -def g(): - h() - -def h(): - i() - -def i(): - j() - -def j(): - k() - -def k(): - l() - -l = lambda: test() - -def test(): - string.capwords(1) - -f() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 07:29:06 2015 From: python-checkins at python.org (nick.coghlan) Date: Thu, 08 Jan 2015 06:29:06 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_474_is_no_longer_deferred?= Message-ID: <20150108062900.8763.85992@psf.io> https://hg.python.org/peps/rev/64d90a1873bf changeset: 5659:64d90a1873bf user: Nick Coghlan date: Thu Jan 08 16:28:51 2015 +1000 summary: PEP 474 is no longer deferred files: pep-0474.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0474.txt b/pep-0474.txt --- a/pep-0474.txt +++ b/pep-0474.txt @@ -3,11 +3,11 @@ Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan -Status: Deferred +Status: Draft Type: Process Content-Type: text/x-rst Created: 19-Jul-2014 -Post-History: 19-Jul-2014 +Post-History: 19-Jul-2014, 08-Jan-2015 Abstract -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Thu Jan 8 07:33:02 2015 From: python-checkins at python.org (nick.coghlan) Date: Thu, 08 Jan 2015 06:33:02 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_462=3A_next_revision_will?= =?utf-8?q?_depend_on_PEP_474?= Message-ID: <20150108063258.125898.91658@psf.io> https://hg.python.org/peps/rev/a1622867aa73 changeset: 5660:a1622867aa73 user: Nick Coghlan date: Thu Jan 08 16:32:49 2015 +1000 summary: PEP 462: next revision will depend on PEP 474 files: pep-0462.txt | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/pep-0462.txt b/pep-0462.txt --- a/pep-0462.txt +++ b/pep-0462.txt @@ -6,6 +6,7 @@ Status: Deferred Type: Process Content-Type: text/x-rst +Requires: 474 Created: 23-Jan-2014 Post-History: 25-Jan-2014, 27-Jan-2014 @@ -25,8 +26,8 @@ PEP Deferral ============ -This PEP is deferred largely because I don't currently have time to work on -it. If anyone would like to take it over, let me know. +This PEP is currently deferred pending updates to redesign it around +the proposal in PEP 474 to create a Kallithea-based forge.python.org service. Rationale for changes to the core development workflow -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Thu Jan 8 07:33:55 2015 From: python-checkins at python.org (nick.coghlan) Date: Thu, 08 Jan 2015 06:33:55 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_474=3A_markup_fixes?= Message-ID: <20150108063342.125896.84081@psf.io> https://hg.python.org/peps/rev/eb2774f9583a changeset: 5661:eb2774f9583a user: Nick Coghlan date: Thu Jan 08 16:33:36 2015 +1000 summary: PEP 474: markup fixes files: pep-0474.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0474.txt b/pep-0474.txt --- a/pep-0474.txt +++ b/pep-0474.txt @@ -175,8 +175,8 @@ in deploying Kallithea using the PSF's Salt based infrastructure automation. * Graham Dumpleton and I have been working on `making it easy - -`__ to deploy + + `__ to deploy demonstration Kallithea instances to the free tier of Red Hat's open source hosting service, OpenShift Online. (See the comments on that post, or the `quickstart issue tracker -- Repository URL: https://hg.python.org/peps From solipsis at pitrou.net Thu Jan 8 10:22:38 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 08 Jan 2015 10:22:38 +0100 Subject: [Python-checkins] Daily reference leaks (e42301eb3ce8): sum=3 Message-ID: results for e42301eb3ce8 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog0MAUR9', '-x'] From python-checkins at python.org Thu Jan 8 12:08:46 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 08 Jan 2015 11:08:46 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogX21h?= =?utf-8?q?ke=5Fssl=5Ftransport=3A_make_the_waiter_parameter_optional?= Message-ID: <20150108110841.8739.63020@psf.io> https://hg.python.org/cpython/rev/f12b845a46c0 changeset: 94078:f12b845a46c0 branch: 3.4 parent: 94076:364e44a49a31 user: Victor Stinner date: Thu Jan 08 12:06:36 2015 +0100 summary: asyncio: _make_ssl_transport: make the waiter parameter optional files: Lib/asyncio/base_events.py | 2 +- Lib/asyncio/selector_events.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -201,7 +201,7 @@ """Create socket transport.""" raise NotImplementedError - def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter, *, + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, server_side=False, server_hostname=None, extra=None, server=None): """Create SSL transport.""" diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -55,7 +55,7 @@ return _SelectorSocketTransport(self, sock, protocol, waiter, extra, server) - def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter, *, + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, server_side=False, server_hostname=None, extra=None, server=None): return _SelectorSslTransport( @@ -165,7 +165,7 @@ else: if sslcontext: self._make_ssl_transport( - conn, protocol_factory(), sslcontext, None, + conn, protocol_factory(), sslcontext, server_side=True, extra={'peername': addr}, server=server) else: self._make_socket_transport( -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 12:08:46 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 08 Jan 2015 11:08:46 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150108110841.8767.74069@psf.io> https://hg.python.org/cpython/rev/a325e2d93c06 changeset: 94079:a325e2d93c06 parent: 94077:58674a027d10 parent: 94078:f12b845a46c0 user: Victor Stinner date: Thu Jan 08 12:07:00 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 2 +- Lib/asyncio/selector_events.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -201,7 +201,7 @@ """Create socket transport.""" raise NotImplementedError - def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter, *, + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, server_side=False, server_hostname=None, extra=None, server=None): """Create SSL transport.""" diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -55,7 +55,7 @@ return _SelectorSocketTransport(self, sock, protocol, waiter, extra, server) - def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter, *, + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, server_side=False, server_hostname=None, extra=None, server=None): return _SelectorSslTransport( @@ -165,7 +165,7 @@ else: if sslcontext: self._make_ssl_transport( - conn, protocol_factory(), sslcontext, None, + conn, protocol_factory(), sslcontext, server_side=True, extra={'peername': addr}, server=server) else: self._make_socket_transport( -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 8 20:10:40 2015 From: python-checkins at python.org (guido.van.rossum) Date: Thu, 08 Jan 2015 19:10:40 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Add_PEP_482=2C_483=2C_484_--_?= =?utf-8?q?type_hints=2E_The_latter_two_are_stubs=2E?= Message-ID: <20150108191038.8741.28350@psf.io> https://hg.python.org/peps/rev/7cbb166b30fd changeset: 5662:7cbb166b30fd user: Guido van Rossum date: Thu Jan 08 11:10:25 2015 -0800 summary: Add PEP 482, 483, 484 -- type hints. The latter two are stubs. files: pep-0482.txt | 222 +++++++++++++++++++++++++++++++++++++++ pep-0483.txt | 28 ++++ pep-0484.txt | 28 ++++ 3 files changed, 278 insertions(+), 0 deletions(-) diff --git a/pep-0482.txt b/pep-0482.txt new file mode 100644 --- /dev/null +++ b/pep-0482.txt @@ -0,0 +1,222 @@ +PEP: 482 +Title: Literature Overview for Type Hinting +Version: $Revision$ +Last-Modified: $Date$ +Author: ?ukasz Langa +Discussions-To: Python-Ideas +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 08-Jan-2015 +Post-History: +Resolution: + +Abstract +======== + +This PEP is one of three related to type hinting. This PEP gives a +literature overview of related work. + + +Existing Approaches in Other Languages +====================================== + +mypy +---- + +(This section is a stub, since mypy [mypy]_ is essentially what we're +proposing.) + +ActionScript +------------ + +ActionScript [actionscript]_ is a class-based, single inheritance, +object-oriented superset of ECMAScript. It supports inferfaces and +strong runtime-checked static typing. Compilation supports a ?strict +dialect? where type mismatches are reported at compile-time. + +Example code with types:: + + package { + import flash.events.Event; + + public class BounceEvent extends Event { + public static const BOUNCE:String = "bounce"; + private var _side:String = "none"; + + public function get side():String { + return _side; + } + + public function BounceEvent(type:String, side:String){ + super(type, true); + _side = side; + } + + public override function clone():Event { + return new BounceEvent(type, _side); + } + } + } + +Dart +---- + +Dart [dart]_ is a class-based, single inheritance, object-oriented +language with C-style syntax. It supports interfaces, abstract classes, +reified generics, and optional typing. + +Types are inferred when possible. The runtime differentiates between two +modes of execution: *checked mode* aimed for development (catching type +errors at runtime) and *production mode* recommended for speed execution +(ignoring types and asserts). + +Example code with types:: + + class Point { + final num x, y; + + Point(this.x, this.y); + + num distanceTo(Point other) { + var dx = x - other.x; + var dy = y - other.y; + return math.sqrt(dx * dx + dy * dy); + } + } + +Hack +---- + +Hack [hack]_ is a programming language that interoperates seamlessly +with PHP. It provides opt-in static type checking, type aliasing, +generics, nullable types, and lambdas. + +Example code with types:: + + alpha(); + } + +TypeScript +---------- + +TypeScript [typescript]_ is a typed superset of JavaScript that adds +interfaces, classes, mixins and modules to the language. + +Type checks are duck typed. Multiple valid function signatures are +specified by supplying overloaded function declarations. Functions and +classes can use generics as type parametrization. Interfaces can have +optional fields. Interfaces can specify array and dictionary types. +Classes can have constructors that implicitly add arguments as fields. +Classes can have static fields. Classes can have private fields. +Classes can have getters/setters for fields (like property). Types are +inferred. + +Example code with types:: + + interface Drivable { + start(): void; + drive(distance: number): boolean; + getPosition(): number; + } + + class Car implements Drivable { + private _isRunning: boolean; + private _distanceFromStart: number; + + constructor() { + this._isRunning = false; + this._distanceFromStart = 0; + } + + public start() { + this._isRunning = true; + } + + public drive(distance: number): boolean { + if (this._isRunning) { + this._distanceFromStart += distance; + return true; + } + return false; + } + + public getPosition(): number { + return this._distanceFromStart; + } + } + + +References +========== + +.. [pep-3107] + http://www.python.org/dev/peps/pep-3107/ + +.. [mypy] + http://mypy-lang.org + +.. [obiwan] + http://pypi.python.org/pypi/obiwan + +.. [numba] + http://numba.pydata.org + +.. [pytypedecl] + https://github.com/google/pytypedecl + +.. [argumentclinic] + https://docs.python.org/3/howto/clinic.html + +.. [numpy] + http://www.numpy.org + +.. [typescript] + http://www.typescriptlang.org + +.. [hack] + http://hacklang.org + +.. [dart] + https://www.dartlang.org + +.. [actionscript] + http://livedocs.adobe.com/specs/actionscript/3/ + +.. [pyflakes] + https://github.com/pyflakes/pyflakes/ + +.. [pylint] + http://www.pylint.org + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0483.txt b/pep-0483.txt new file mode 100644 --- /dev/null +++ b/pep-0483.txt @@ -0,0 +1,28 @@ +PEP: 483 +Title: The Theory of Type Hinting +Version: $Revision$ +Last-Modified: $Date$ +Author: Guido van Rossum +Discussions-To: Python-Ideas +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 08-Jan-2015 +Post-History: +Resolution: + +Abstract +======== + +This PEP is currently a stub. The content should be copied from +https://quip.com/r69HA9GhGa7J and reformatted. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0484.txt b/pep-0484.txt new file mode 100644 --- /dev/null +++ b/pep-0484.txt @@ -0,0 +1,28 @@ +PEP: 484 +Title: Type Hints +Version: $Revision$ +Last-Modified: $Date$ +Author: Guido van Rossum , Jukka Lehtosalo , ?ukasz Langa +Discussions-To: Python-Ideas +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 08-Jan-2015 +Post-History: +Resolution: + +Abstract +======== + +This PEP is currently a stub. The content should be copied from +https://github.com/ambv/typehinting (but omitting the literature overview, which is PEP 482) and reformatted. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 9 00:11:33 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 08 Jan 2015 23:11:33 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogVHJ1?= =?utf-8?q?ncate_to_80_columns?= Message-ID: <20150108231129.22395.3850@psf.io> https://hg.python.org/cpython/rev/756d09136a2c changeset: 94080:756d09136a2c branch: 3.4 parent: 94078:f12b845a46c0 user: Victor Stinner date: Fri Jan 09 00:09:10 2015 +0100 summary: asyncio: Truncate to 80 columns files: Lib/asyncio/base_events.py | 4 +- Lib/asyncio/coroutines.py | 12 +++- Lib/asyncio/selector_events.py | 7 +- Lib/asyncio/tasks.py | 2 +- Lib/asyncio/unix_events.py | 3 +- Lib/asyncio/windows_events.py | 3 +- Lib/asyncio/windows_utils.py | 7 +- Lib/test/test_asyncio/test_base_events.py | 9 ++- Lib/test/test_asyncio/test_futures.py | 24 +++++++--- Lib/test/test_asyncio/test_streams.py | 6 +- Lib/test/test_asyncio/test_subprocess.py | 4 +- Lib/test/test_asyncio/test_tasks.py | 12 +++- 12 files changed, 60 insertions(+), 33 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -201,8 +201,8 @@ """Create socket transport.""" raise NotImplementedError - def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, - server_side=False, server_hostname=None, + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, + *, server_side=False, server_hostname=None, extra=None, server=None): """Create SSL transport.""" raise NotImplementedError diff --git a/Lib/asyncio/coroutines.py b/Lib/asyncio/coroutines.py --- a/Lib/asyncio/coroutines.py +++ b/Lib/asyncio/coroutines.py @@ -182,14 +182,18 @@ and not inspect.isgeneratorfunction(coro.func)): filename, lineno = events._get_function_source(coro.func) if coro.gi_frame is None: - coro_repr = '%s() done, defined at %s:%s' % (coro_name, filename, lineno) + coro_repr = ('%s() done, defined at %s:%s' + % (coro_name, filename, lineno)) else: - coro_repr = '%s() running, defined at %s:%s' % (coro_name, filename, lineno) + coro_repr = ('%s() running, defined at %s:%s' + % (coro_name, filename, lineno)) elif coro.gi_frame is not None: lineno = coro.gi_frame.f_lineno - coro_repr = '%s() running at %s:%s' % (coro_name, filename, lineno) + coro_repr = ('%s() running at %s:%s' + % (coro_name, filename, lineno)) else: lineno = coro.gi_code.co_firstlineno - coro_repr = '%s() done, defined at %s:%s' % (coro_name, filename, lineno) + coro_repr = ('%s() done, defined at %s:%s' + % (coro_name, filename, lineno)) return coro_repr diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -55,8 +55,8 @@ return _SelectorSocketTransport(self, sock, protocol, waiter, extra, server) - def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, - server_side=False, server_hostname=None, + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, + *, server_side=False, server_hostname=None, extra=None, server=None): return _SelectorSslTransport( self, rawsock, protocol, sslcontext, waiter, @@ -484,7 +484,8 @@ info.append('read=idle') polling = _test_selector_event(self._loop._selector, - self._sock_fd, selectors.EVENT_WRITE) + self._sock_fd, + selectors.EVENT_WRITE) if polling: state = 'polling' else: diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -68,7 +68,7 @@ return {t for t in cls._all_tasks if t._loop is loop} def __init__(self, coro, *, loop=None): - assert coroutines.iscoroutine(coro), repr(coro) # Not a coroutine function! + assert coroutines.iscoroutine(coro), repr(coro) super().__init__(loop=loop) if self._source_traceback: del self._source_traceback[-1] diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -69,7 +69,8 @@ """ if (coroutines.iscoroutine(callback) or coroutines.iscoroutinefunction(callback)): - raise TypeError("coroutines cannot be used with add_signal_handler()") + raise TypeError("coroutines cannot be used " + "with add_signal_handler()") self._check_signal(sig) self._check_closed() try: diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -424,7 +424,8 @@ else: return windows_utils.PipeHandle(handle) - return self._register(ov, None, finish_connect_pipe, wait_for_post=True) + return self._register(ov, None, finish_connect_pipe, + wait_for_post=True) def wait_for_handle(self, handle, timeout=None): """Wait for a handle. diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -36,15 +36,16 @@ def socketpair(family=socket.AF_INET, type=socket.SOCK_STREAM, proto=0): """A socket pair usable as a self-pipe, for Windows. - Origin: https://gist.github.com/4325783, by Geert Jansen. Public domain. + Origin: https://gist.github.com/4325783, by Geert Jansen. + Public domain. """ if family == socket.AF_INET: host = '127.0.0.1' elif family == socket.AF_INET6: host = '::1' else: - raise ValueError("Only AF_INET and AF_INET6 socket address families " - "are supported") + raise ValueError("Only AF_INET and AF_INET6 socket address " + "families are supported") if type != socket.SOCK_STREAM: raise ValueError("Only SOCK_STREAM socket type is supported") if proto != 0: diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -285,7 +285,8 @@ @mock.patch('asyncio.base_events.logger') def test__run_once_logging(self, m_logger): def slow_select(timeout): - # Sleep a bit longer than a second to avoid timer resolution issues. + # Sleep a bit longer than a second to avoid timer resolution + # issues. time.sleep(1.1) return [] @@ -1217,14 +1218,16 @@ self.loop.run_forever() fmt, *args = m_logger.warning.call_args[0] self.assertRegex(fmt % tuple(args), - "^Executing took .* seconds$") + "^Executing " + "took .* seconds$") # slow task asyncio.async(stop_loop_coro(self.loop), loop=self.loop) self.loop.run_forever() fmt, *args = m_logger.warning.call_args[0] self.assertRegex(fmt % tuple(args), - "^Executing took .* seconds$") + "^Executing " + "took .* seconds$") if __name__ == '__main__': diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py --- a/Lib/test/test_asyncio/test_futures.py +++ b/Lib/test/test_asyncio/test_futures.py @@ -133,7 +133,8 @@ exc = RuntimeError() f_exception = asyncio.Future(loop=self.loop) f_exception.set_exception(exc) - self.assertEqual(repr(f_exception), '') + self.assertEqual(repr(f_exception), + '') self.assertIs(f_exception.exception(), exc) def func_repr(func): @@ -332,16 +333,21 @@ if debug: frame = source_traceback[-1] regex = (r'^Future exception was never retrieved\n' - r'future: \n' - r'source_traceback: Object created at \(most recent call last\):\n' + r'future: \n' + r'source_traceback: Object ' + r'created at \(most recent call last\):\n' r' File' r'.*\n' - r' File "{filename}", line {lineno}, in check_future_exception_never_retrieved\n' + r' File "{filename}", line {lineno}, ' + r'in check_future_exception_never_retrieved\n' r' future = asyncio\.Future\(loop=self\.loop\)$' - ).format(filename=re.escape(frame[0]), lineno=frame[1]) + ).format(filename=re.escape(frame[0]), + lineno=frame[1]) else: regex = (r'^Future exception was never retrieved\n' - r'future: $' + r'future: ' + r'$' ) exc_info = (type(exc), exc, exc.__traceback__) m_log.error.assert_called_once_with(mock.ANY, exc_info=exc_info) @@ -352,12 +358,14 @@ r'Future/Task created at \(most recent call last\):\n' r' File' r'.*\n' - r' File "{filename}", line {lineno}, in check_future_exception_never_retrieved\n' + r' File "{filename}", line {lineno}, ' + r'in check_future_exception_never_retrieved\n' r' future = asyncio\.Future\(loop=self\.loop\)\n' r'Traceback \(most recent call last\):\n' r'.*\n' r'MemoryError$' - ).format(filename=re.escape(frame[0]), lineno=frame[1]) + ).format(filename=re.escape(frame[0]), + lineno=frame[1]) else: regex = (r'^Future/Task exception was never retrieved\n' r'Traceback \(most recent call last\):\n' diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -613,8 +613,10 @@ watcher.attach_loop(self.loop) try: asyncio.set_child_watcher(watcher) - proc = self.loop.run_until_complete( - asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop)) + create = asyncio.create_subprocess_exec(*args, + pass_fds={wfd}, + loop=self.loop) + proc = self.loop.run_until_complete(create) self.loop.run_until_complete(proc.wait()) finally: asyncio.set_child_watcher(None) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -115,7 +115,9 @@ def test_send_signal(self): code = 'import time; print("sleeping", flush=True); time.sleep(3600)' args = [sys.executable, '-c', code] - create = asyncio.create_subprocess_exec(*args, loop=self.loop, stdout=subprocess.PIPE) + create = asyncio.create_subprocess_exec(*args, + stdout=subprocess.PIPE, + loop=self.loop) proc = self.loop.run_until_complete(create) @asyncio.coroutine diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -208,7 +208,8 @@ self.assertEqual(notmuch.__name__, 'notmuch') if PY35: self.assertEqual(notmuch.__qualname__, - 'TaskTests.test_task_repr_coro_decorator..notmuch') + 'TaskTests.test_task_repr_coro_decorator' + '..notmuch') self.assertEqual(notmuch.__module__, __name__) # test coroutine object @@ -218,7 +219,8 @@ # function, as expected, and have a qualified name (__qualname__ # attribute). coro_name = 'notmuch' - coro_qualname = 'TaskTests.test_task_repr_coro_decorator..notmuch' + coro_qualname = ('TaskTests.test_task_repr_coro_decorator' + '..notmuch') else: # On Python < 3.5, generators inherit the name of the code, not of # the function. See: http://bugs.python.org/issue21205 @@ -239,7 +241,8 @@ else: code = gen.gi_code coro = ('%s() running at %s:%s' - % (coro_qualname, code.co_filename, code.co_firstlineno)) + % (coro_qualname, code.co_filename, + code.co_firstlineno)) self.assertEqual(repr(gen), '' % coro) @@ -1678,7 +1681,8 @@ self.assertTrue(m_log.error.called) message = m_log.error.call_args[0][0] func_filename, func_lineno = test_utils.get_function_source(coro_noop) - regex = (r'^ was never yielded from\n' + regex = (r'^ ' + r'was never yielded from\n' r'Coroutine object created at \(most recent call last\):\n' r'.*\n' r' File "%s", line %s, in test_coroutine_never_yielded\n' -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 00:11:33 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 08 Jan 2015 23:11:33 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150108231129.72579.48016@psf.io> https://hg.python.org/cpython/rev/c644d49130b6 changeset: 94081:c644d49130b6 parent: 94079:a325e2d93c06 parent: 94080:756d09136a2c user: Victor Stinner date: Fri Jan 09 00:09:35 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 4 +- Lib/asyncio/coroutines.py | 12 +++- Lib/asyncio/selector_events.py | 7 +- Lib/asyncio/tasks.py | 2 +- Lib/asyncio/unix_events.py | 3 +- Lib/asyncio/windows_events.py | 3 +- Lib/asyncio/windows_utils.py | 7 +- Lib/test/test_asyncio/test_base_events.py | 9 ++- Lib/test/test_asyncio/test_futures.py | 24 +++++++--- Lib/test/test_asyncio/test_streams.py | 6 +- Lib/test/test_asyncio/test_subprocess.py | 4 +- Lib/test/test_asyncio/test_tasks.py | 12 +++- 12 files changed, 60 insertions(+), 33 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -201,8 +201,8 @@ """Create socket transport.""" raise NotImplementedError - def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, - server_side=False, server_hostname=None, + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, + *, server_side=False, server_hostname=None, extra=None, server=None): """Create SSL transport.""" raise NotImplementedError diff --git a/Lib/asyncio/coroutines.py b/Lib/asyncio/coroutines.py --- a/Lib/asyncio/coroutines.py +++ b/Lib/asyncio/coroutines.py @@ -182,14 +182,18 @@ and not inspect.isgeneratorfunction(coro.func)): filename, lineno = events._get_function_source(coro.func) if coro.gi_frame is None: - coro_repr = '%s() done, defined at %s:%s' % (coro_name, filename, lineno) + coro_repr = ('%s() done, defined at %s:%s' + % (coro_name, filename, lineno)) else: - coro_repr = '%s() running, defined at %s:%s' % (coro_name, filename, lineno) + coro_repr = ('%s() running, defined at %s:%s' + % (coro_name, filename, lineno)) elif coro.gi_frame is not None: lineno = coro.gi_frame.f_lineno - coro_repr = '%s() running at %s:%s' % (coro_name, filename, lineno) + coro_repr = ('%s() running at %s:%s' + % (coro_name, filename, lineno)) else: lineno = coro.gi_code.co_firstlineno - coro_repr = '%s() done, defined at %s:%s' % (coro_name, filename, lineno) + coro_repr = ('%s() done, defined at %s:%s' + % (coro_name, filename, lineno)) return coro_repr diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -55,8 +55,8 @@ return _SelectorSocketTransport(self, sock, protocol, waiter, extra, server) - def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, - server_side=False, server_hostname=None, + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, + *, server_side=False, server_hostname=None, extra=None, server=None): return _SelectorSslTransport( self, rawsock, protocol, sslcontext, waiter, @@ -484,7 +484,8 @@ info.append('read=idle') polling = _test_selector_event(self._loop._selector, - self._sock_fd, selectors.EVENT_WRITE) + self._sock_fd, + selectors.EVENT_WRITE) if polling: state = 'polling' else: diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -68,7 +68,7 @@ return {t for t in cls._all_tasks if t._loop is loop} def __init__(self, coro, *, loop=None): - assert coroutines.iscoroutine(coro), repr(coro) # Not a coroutine function! + assert coroutines.iscoroutine(coro), repr(coro) super().__init__(loop=loop) if self._source_traceback: del self._source_traceback[-1] diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -69,7 +69,8 @@ """ if (coroutines.iscoroutine(callback) or coroutines.iscoroutinefunction(callback)): - raise TypeError("coroutines cannot be used with add_signal_handler()") + raise TypeError("coroutines cannot be used " + "with add_signal_handler()") self._check_signal(sig) self._check_closed() try: diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -424,7 +424,8 @@ else: return windows_utils.PipeHandle(handle) - return self._register(ov, None, finish_connect_pipe, wait_for_post=True) + return self._register(ov, None, finish_connect_pipe, + wait_for_post=True) def wait_for_handle(self, handle, timeout=None): """Wait for a handle. diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -36,15 +36,16 @@ def socketpair(family=socket.AF_INET, type=socket.SOCK_STREAM, proto=0): """A socket pair usable as a self-pipe, for Windows. - Origin: https://gist.github.com/4325783, by Geert Jansen. Public domain. + Origin: https://gist.github.com/4325783, by Geert Jansen. + Public domain. """ if family == socket.AF_INET: host = '127.0.0.1' elif family == socket.AF_INET6: host = '::1' else: - raise ValueError("Only AF_INET and AF_INET6 socket address families " - "are supported") + raise ValueError("Only AF_INET and AF_INET6 socket address " + "families are supported") if type != socket.SOCK_STREAM: raise ValueError("Only SOCK_STREAM socket type is supported") if proto != 0: diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -285,7 +285,8 @@ @mock.patch('asyncio.base_events.logger') def test__run_once_logging(self, m_logger): def slow_select(timeout): - # Sleep a bit longer than a second to avoid timer resolution issues. + # Sleep a bit longer than a second to avoid timer resolution + # issues. time.sleep(1.1) return [] @@ -1217,14 +1218,16 @@ self.loop.run_forever() fmt, *args = m_logger.warning.call_args[0] self.assertRegex(fmt % tuple(args), - "^Executing took .* seconds$") + "^Executing " + "took .* seconds$") # slow task asyncio.async(stop_loop_coro(self.loop), loop=self.loop) self.loop.run_forever() fmt, *args = m_logger.warning.call_args[0] self.assertRegex(fmt % tuple(args), - "^Executing took .* seconds$") + "^Executing " + "took .* seconds$") if __name__ == '__main__': diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py --- a/Lib/test/test_asyncio/test_futures.py +++ b/Lib/test/test_asyncio/test_futures.py @@ -133,7 +133,8 @@ exc = RuntimeError() f_exception = asyncio.Future(loop=self.loop) f_exception.set_exception(exc) - self.assertEqual(repr(f_exception), '') + self.assertEqual(repr(f_exception), + '') self.assertIs(f_exception.exception(), exc) def func_repr(func): @@ -332,16 +333,21 @@ if debug: frame = source_traceback[-1] regex = (r'^Future exception was never retrieved\n' - r'future: \n' - r'source_traceback: Object created at \(most recent call last\):\n' + r'future: \n' + r'source_traceback: Object ' + r'created at \(most recent call last\):\n' r' File' r'.*\n' - r' File "{filename}", line {lineno}, in check_future_exception_never_retrieved\n' + r' File "{filename}", line {lineno}, ' + r'in check_future_exception_never_retrieved\n' r' future = asyncio\.Future\(loop=self\.loop\)$' - ).format(filename=re.escape(frame[0]), lineno=frame[1]) + ).format(filename=re.escape(frame[0]), + lineno=frame[1]) else: regex = (r'^Future exception was never retrieved\n' - r'future: $' + r'future: ' + r'$' ) exc_info = (type(exc), exc, exc.__traceback__) m_log.error.assert_called_once_with(mock.ANY, exc_info=exc_info) @@ -352,12 +358,14 @@ r'Future/Task created at \(most recent call last\):\n' r' File' r'.*\n' - r' File "{filename}", line {lineno}, in check_future_exception_never_retrieved\n' + r' File "{filename}", line {lineno}, ' + r'in check_future_exception_never_retrieved\n' r' future = asyncio\.Future\(loop=self\.loop\)\n' r'Traceback \(most recent call last\):\n' r'.*\n' r'MemoryError$' - ).format(filename=re.escape(frame[0]), lineno=frame[1]) + ).format(filename=re.escape(frame[0]), + lineno=frame[1]) else: regex = (r'^Future/Task exception was never retrieved\n' r'Traceback \(most recent call last\):\n' diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -613,8 +613,10 @@ watcher.attach_loop(self.loop) try: asyncio.set_child_watcher(watcher) - proc = self.loop.run_until_complete( - asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop)) + create = asyncio.create_subprocess_exec(*args, + pass_fds={wfd}, + loop=self.loop) + proc = self.loop.run_until_complete(create) self.loop.run_until_complete(proc.wait()) finally: asyncio.set_child_watcher(None) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -115,7 +115,9 @@ def test_send_signal(self): code = 'import time; print("sleeping", flush=True); time.sleep(3600)' args = [sys.executable, '-c', code] - create = asyncio.create_subprocess_exec(*args, loop=self.loop, stdout=subprocess.PIPE) + create = asyncio.create_subprocess_exec(*args, + stdout=subprocess.PIPE, + loop=self.loop) proc = self.loop.run_until_complete(create) @asyncio.coroutine diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -208,7 +208,8 @@ self.assertEqual(notmuch.__name__, 'notmuch') if PY35: self.assertEqual(notmuch.__qualname__, - 'TaskTests.test_task_repr_coro_decorator..notmuch') + 'TaskTests.test_task_repr_coro_decorator' + '..notmuch') self.assertEqual(notmuch.__module__, __name__) # test coroutine object @@ -218,7 +219,8 @@ # function, as expected, and have a qualified name (__qualname__ # attribute). coro_name = 'notmuch' - coro_qualname = 'TaskTests.test_task_repr_coro_decorator..notmuch' + coro_qualname = ('TaskTests.test_task_repr_coro_decorator' + '..notmuch') else: # On Python < 3.5, generators inherit the name of the code, not of # the function. See: http://bugs.python.org/issue21205 @@ -239,7 +241,8 @@ else: code = gen.gi_code coro = ('%s() running at %s:%s' - % (coro_qualname, code.co_filename, code.co_firstlineno)) + % (coro_qualname, code.co_filename, + code.co_firstlineno)) self.assertEqual(repr(gen), '' % coro) @@ -1678,7 +1681,8 @@ self.assertTrue(m_log.error.called) message = m_log.error.call_args[0][0] func_filename, func_lineno = test_utils.get_function_source(coro_noop) - regex = (r'^ was never yielded from\n' + regex = (r'^ ' + r'was never yielded from\n' r'Coroutine object created at \(most recent call last\):\n' r'.*\n' r' File "%s", line %s, in test_coroutine_never_yielded\n' -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 00:14:04 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 08 Jan 2015 23:14:04 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_selectors=3A_truncate_to_8?= =?utf-8?q?0_characters?= Message-ID: <20150108231350.125890.26512@psf.io> https://hg.python.org/cpython/rev/e86d0ef45e21 changeset: 94082:e86d0ef45e21 user: Victor Stinner date: Fri Jan 09 00:13:39 2015 +0100 summary: selectors: truncate to 80 characters files: Lib/selectors.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -576,7 +576,8 @@ super().close() -# Choose the best implementation: roughly, epoll|kqueue|devpoll > poll > select. +# Choose the best implementation, roughly: +# epoll|kqueue|devpoll > poll > select. # select() also can't accept a FD > FD_SETSIZE (usually around 1024) if 'KqueueSelector' in globals(): DefaultSelector = KqueueSelector -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 01:32:40 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 00:32:40 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?= Message-ID: <20150109003233.72579.89276@psf.io> https://hg.python.org/cpython/rev/94ee2fdf2df3 changeset: 94084:94ee2fdf2df3 parent: 94082:e86d0ef45e21 parent: 94083:a04d87fbff4e user: Victor Stinner date: Fri Jan 09 01:32:25 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-dev.rst | 5 ++++ Doc/library/asyncio-subprocess.rst | 19 ++++++++++++++++++ 2 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -74,6 +74,11 @@ The :ref:`Synchronization primitives ` section describes ways to synchronize tasks. + The :ref:`Subprocess and threads ` section lists + asyncio limitations to run subprocesses from different threads. + + + .. _asyncio-handle-blocking: diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -297,6 +297,25 @@ ``N`` (Unix only). +.. _asyncio-subprocess-threads: + +Subprocess and threads +====================== + +asyncio supports running subprocesses from different threads, but there +are limits: + +* An event loop must run in the main thread +* The child watcher must be instantiated in the main thread, before executing + subprocesses from other threads. Call the :func:`get_child_watcher` + function in the main thread to instantiate the child watcher. + +.. seealso:: + + The :ref:`Concurrency and multithreading in asyncio + ` section. + + Subprocess examples =================== -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 01:32:40 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 00:32:40 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbyBkb2M6?= =?utf-8?q?_list_limitations_to_run_subprocesses_from_different_threads?= Message-ID: <20150109003233.72575.26860@psf.io> https://hg.python.org/cpython/rev/a04d87fbff4e changeset: 94083:a04d87fbff4e branch: 3.4 parent: 94080:756d09136a2c user: Victor Stinner date: Fri Jan 09 01:32:02 2015 +0100 summary: asyncio doc: list limitations to run subprocesses from different threads files: Doc/library/asyncio-dev.rst | 5 ++++ Doc/library/asyncio-subprocess.rst | 19 ++++++++++++++++++ 2 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -74,6 +74,11 @@ The :ref:`Synchronization primitives ` section describes ways to synchronize tasks. + The :ref:`Subprocess and threads ` section lists + asyncio limitations to run subprocesses from different threads. + + + .. _asyncio-handle-blocking: diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -297,6 +297,25 @@ ``N`` (Unix only). +.. _asyncio-subprocess-threads: + +Subprocess and threads +====================== + +asyncio supports running subprocesses from different threads, but there +are limits: + +* An event loop must run in the main thread +* The child watcher must be instantiated in the main thread, before executing + subprocesses from other threads. Call the :func:`get_child_watcher` + function in the main thread to instantiate the child watcher. + +.. seealso:: + + The :ref:`Concurrency and multithreading in asyncio + ` section. + + Subprocess examples =================== -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 01:44:04 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 00:44:04 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogc3lu?= =?utf-8?q?c_with_Tulip?= Message-ID: <20150109004400.22395.3543@psf.io> https://hg.python.org/cpython/rev/d07386812daa changeset: 94085:d07386812daa branch: 3.4 parent: 94083:a04d87fbff4e user: Victor Stinner date: Fri Jan 09 01:42:52 2015 +0100 summary: asyncio: sync with Tulip * Document why set_result() calls are safe * Cleanup gather(). Use public methods instead of hacks to consume the exception of a future. * sock_connect(): pass directly the fd to _sock_connect_done instead of the socket. files: Lib/asyncio/queues.py | 6 ++++++ Lib/asyncio/selector_events.py | 6 +++--- Lib/asyncio/tasks.py | 11 +++++++---- 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py --- a/Lib/asyncio/queues.py +++ b/Lib/asyncio/queues.py @@ -126,6 +126,8 @@ # Use _put and _get instead of passing item straight to getter, in # case a subclass has logic that must run (e.g. JoinableQueue). self._put(item) + + # getter cannot be cancelled, we just removed done getters getter.set_result(self._get()) elif self._maxsize > 0 and self._maxsize <= self.qsize(): @@ -152,6 +154,8 @@ # Use _put and _get instead of passing item straight to getter, in # case a subclass has logic that must run (e.g. JoinableQueue). self._put(item) + + # getter cannot be cancelled, we just removed done getters getter.set_result(self._get()) elif self._maxsize > 0 and self._maxsize <= self.qsize(): @@ -200,6 +204,8 @@ item, putter = self._putters.popleft() self._put(item) # Wake putter on next tick. + + # getter cannot be cancelled, we just removed done putters putter.set_result(None) return self._get() diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -363,15 +363,15 @@ break except BlockingIOError: fut.add_done_callback(functools.partial(self._sock_connect_done, - sock)) + fd)) self.add_writer(fd, self._sock_connect_cb, fut, sock, address) except Exception as exc: fut.set_exception(exc) else: fut.set_result(None) - def _sock_connect_done(self, sock, fut): - self.remove_writer(sock.fileno()) + def _sock_connect_done(self, fd, fut): + self.remove_writer(fd) def _sock_connect_cb(self, fut, sock, address): if fut.cancelled(): diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -582,11 +582,12 @@ def _done_callback(i, fut): nonlocal nfinished - if outer._state != futures._PENDING: - if fut._exception is not None: + if outer.done(): + if not fut.cancelled(): # Mark exception retrieved. fut.exception() return + if fut._state == futures._CANCELLED: res = futures.CancelledError() if not return_exceptions: @@ -644,9 +645,11 @@ def _done_callback(inner): if outer.cancelled(): - # Mark inner's result as retrieved. - inner.cancelled() or inner.exception() + if not inner.cancelled(): + # Mark inner's result as retrieved. + inner.exception() return + if inner.cancelled(): outer.cancel() else: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 01:44:04 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 00:44:04 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150109004400.72565.50204@psf.io> https://hg.python.org/cpython/rev/f7aef9f3daef changeset: 94086:f7aef9f3daef parent: 94084:94ee2fdf2df3 parent: 94085:d07386812daa user: Victor Stinner date: Fri Jan 09 01:43:04 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/queues.py | 6 ++++++ Lib/asyncio/selector_events.py | 6 +++--- Lib/asyncio/tasks.py | 11 +++++++---- 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py --- a/Lib/asyncio/queues.py +++ b/Lib/asyncio/queues.py @@ -126,6 +126,8 @@ # Use _put and _get instead of passing item straight to getter, in # case a subclass has logic that must run (e.g. JoinableQueue). self._put(item) + + # getter cannot be cancelled, we just removed done getters getter.set_result(self._get()) elif self._maxsize > 0 and self._maxsize <= self.qsize(): @@ -152,6 +154,8 @@ # Use _put and _get instead of passing item straight to getter, in # case a subclass has logic that must run (e.g. JoinableQueue). self._put(item) + + # getter cannot be cancelled, we just removed done getters getter.set_result(self._get()) elif self._maxsize > 0 and self._maxsize <= self.qsize(): @@ -200,6 +204,8 @@ item, putter = self._putters.popleft() self._put(item) # Wake putter on next tick. + + # getter cannot be cancelled, we just removed done putters putter.set_result(None) return self._get() diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -363,15 +363,15 @@ break except BlockingIOError: fut.add_done_callback(functools.partial(self._sock_connect_done, - sock)) + fd)) self.add_writer(fd, self._sock_connect_cb, fut, sock, address) except Exception as exc: fut.set_exception(exc) else: fut.set_result(None) - def _sock_connect_done(self, sock, fut): - self.remove_writer(sock.fileno()) + def _sock_connect_done(self, fd, fut): + self.remove_writer(fd) def _sock_connect_cb(self, fut, sock, address): if fut.cancelled(): diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -582,11 +582,12 @@ def _done_callback(i, fut): nonlocal nfinished - if outer._state != futures._PENDING: - if fut._exception is not None: + if outer.done(): + if not fut.cancelled(): # Mark exception retrieved. fut.exception() return + if fut._state == futures._CANCELLED: res = futures.CancelledError() if not return_exceptions: @@ -644,9 +645,11 @@ def _done_callback(inner): if outer.cancelled(): - # Mark inner's result as retrieved. - inner.cancelled() or inner.exception() + if not inner.cancelled(): + # Mark inner's result as retrieved. + inner.exception() return + if inner.cancelled(): outer.cancel() else: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 02:14:19 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 01:14:19 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322038=3A_pyatomic?= =?utf-8?q?=2Eh_now_uses_stdatomic=2Eh_or_GCC_built-in_functions_for?= Message-ID: <20150109011411.8765.83201@psf.io> https://hg.python.org/cpython/rev/fbe87fb071a6 changeset: 94087:fbe87fb071a6 user: Victor Stinner date: Fri Jan 09 02:13:19 2015 +0100 summary: Issue #22038: pyatomic.h now uses stdatomic.h or GCC built-in functions for atomic memory access if available. Patch written by Vitor de Lima and Gustavo Temple. files: Include/pyatomic.h | 80 ++++++++++++++++++++++++++++++++- Misc/ACKS | 2 + Misc/NEWS | 4 + configure | 65 +++++++++++++++++++++++++++ configure.ac | 39 ++++++++++++++++ pyconfig.h.in | 6 ++ 6 files changed, 193 insertions(+), 3 deletions(-) diff --git a/Include/pyatomic.h b/Include/pyatomic.h --- a/Include/pyatomic.h +++ b/Include/pyatomic.h @@ -1,12 +1,15 @@ #ifndef Py_LIMITED_API #ifndef Py_ATOMIC_H #define Py_ATOMIC_H -/* XXX: When compilers start offering a stdatomic.h with lock-free - atomic_int and atomic_address types, include that here and rewrite - the atomic operations in terms of it. */ #include "dynamic_annotations.h" +#include "pyconfig.h" + +#if defined(HAVE_STD_ATOMIC) +#include +#endif + #ifdef __cplusplus extern "C" { #endif @@ -20,6 +23,76 @@ * Beware, the implementations here are deep magic. */ +#if defined(HAVE_STD_ATOMIC) + +typedef enum _Py_memory_order { + _Py_memory_order_relaxed = memory_order_relaxed, + _Py_memory_order_acquire = memory_order_acquire, + _Py_memory_order_release = memory_order_release, + _Py_memory_order_acq_rel = memory_order_acq_rel, + _Py_memory_order_seq_cst = memory_order_seq_cst +} _Py_memory_order; + +typedef struct _Py_atomic_address { + _Atomic void *_value; +} _Py_atomic_address; + +typedef struct _Py_atomic_int { + atomic_int _value; +} _Py_atomic_int; + +#define _Py_atomic_signal_fence(/*memory_order*/ ORDER) \ + atomic_signal_fence(ORDER) + +#define _Py_atomic_thread_fence(/*memory_order*/ ORDER) \ + atomic_thread_fence(ORDER) + +#define _Py_atomic_store_explicit(ATOMIC_VAL, NEW_VAL, ORDER) \ + atomic_store_explicit(&(ATOMIC_VAL)->_value, NEW_VAL, ORDER) + +#define _Py_atomic_load_explicit(ATOMIC_VAL, ORDER) \ + atomic_load_explicit(&(ATOMIC_VAL)->_value, ORDER) + +/* Use builtin atomic operations in GCC >= 4.7 */ +#elif defined(HAVE_BUILTIN_ATOMIC) + +typedef enum _Py_memory_order { + _Py_memory_order_relaxed = __ATOMIC_RELAXED, + _Py_memory_order_acquire = __ATOMIC_ACQUIRE, + _Py_memory_order_release = __ATOMIC_RELEASE, + _Py_memory_order_acq_rel = __ATOMIC_ACQ_REL, + _Py_memory_order_seq_cst = __ATOMIC_SEQ_CST +} _Py_memory_order; + +typedef struct _Py_atomic_address { + void *_value; +} _Py_atomic_address; + +typedef struct _Py_atomic_int { + int _value; +} _Py_atomic_int; + +#define _Py_atomic_signal_fence(/*memory_order*/ ORDER) \ + __atomic_signal_fence(ORDER) + +#define _Py_atomic_thread_fence(/*memory_order*/ ORDER) \ + __atomic_thread_fence(ORDER) + +#define _Py_atomic_store_explicit(ATOMIC_VAL, NEW_VAL, ORDER) \ + (assert((ORDER) == __ATOMIC_RELAXED \ + || (ORDER) == __ATOMIC_SEQ_CST \ + || (ORDER) == __ATOMIC_RELEASE), \ + __atomic_store_n(&(ATOMIC_VAL)->_value, NEW_VAL, ORDER)) + +#define _Py_atomic_load_explicit(ATOMIC_VAL, ORDER) \ + (assert((ORDER) == __ATOMIC_RELAXED \ + || (ORDER) == __ATOMIC_SEQ_CST \ + || (ORDER) == __ATOMIC_ACQUIRE \ + || (ORDER) == __ATOMIC_CONSUME), \ + __atomic_load_n(&(ATOMIC_VAL)->_value, ORDER)) + +#else + typedef enum _Py_memory_order { _Py_memory_order_relaxed, _Py_memory_order_acquire, @@ -162,6 +235,7 @@ ((ATOMIC_VAL)->_value) #endif /* !gcc x86 */ +#endif /* Standardized shortcuts. */ #define _Py_atomic_store(ATOMIC_VAL, NEW_VAL) \ diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -821,6 +821,7 @@ Shawn Ligocki Martin Ligr Gediminas Liktaras +Vitor de Lima Grant Limberg Christopher Lindblad Ulf A. Lindgren @@ -1355,6 +1356,7 @@ Amy Taylor Monty Taylor Anatoly Techtonik +Gustavo Temple Mikhail Terekhov Victor Terr?n Richard M. Tew diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #22038: pyatomic.h now uses stdatomic.h or GCC built-in functions for + atomic memory access if available. Patch written by Vitor de Lima and Gustavo + Temple. + - Issue #23048: Fix jumping out of an infinite while loop in the pdb. - Issue #20335: bytes constructor now raises TypeError when encoding or errors diff --git a/configure b/configure --- a/configure +++ b/configure @@ -15703,6 +15703,71 @@ esac fi +# Check for stdatomic.h +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for stdatomic.h" >&5 +$as_echo_n "checking for stdatomic.h... " >&6; } +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + + + #include + _Atomic int value = ATOMIC_VAR_INIT(1); + int main() { + int loaded_value = atomic_load(&value); + return 0; + } + + +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + have_stdatomic_h=yes +else + have_stdatomic_h=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $have_stdatomic_h" >&5 +$as_echo "$have_stdatomic_h" >&6; } + +if test "$have_stdatomic_h" = yes; then + +$as_echo "#define HAVE_STD_ATOMIC 1" >>confdefs.h + +fi + +# Check for GCC >= 4.7 __atomic builtins +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for GCC >= 4.7 __atomic builtins" >&5 +$as_echo_n "checking for GCC >= 4.7 __atomic builtins... " >&6; } +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + + + volatile int val = 1; + int main() { + __atomic_load_n(&val, __ATOMIC_SEQ_CST); + return 0; + } + + +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + have_builtin_atomic=yes +else + have_builtin_atomic=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $have_builtin_atomic" >&5 +$as_echo "$have_builtin_atomic" >&6; } + +if test "$have_builtin_atomic" = yes; then + +$as_echo "#define HAVE_BUILTIN_ATOMIC 1" >>confdefs.h + +fi + # ensurepip option { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ensurepip" >&5 $as_echo_n "checking for ensurepip... " >&6; } diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -4884,6 +4884,45 @@ esac fi +# Check for stdatomic.h +AC_MSG_CHECKING(for stdatomic.h) +AC_LINK_IFELSE( +[ + AC_LANG_SOURCE([[ + #include + _Atomic int value = ATOMIC_VAR_INIT(1); + int main() { + int loaded_value = atomic_load(&value); + return 0; + } + ]]) +],[have_stdatomic_h=yes],[have_stdatomic_h=no]) + +AC_MSG_RESULT($have_stdatomic_h) + +if test "$have_stdatomic_h" = yes; then + AC_DEFINE(HAVE_STD_ATOMIC, 1, [Has stdatomic.h]) +fi + +# Check for GCC >= 4.7 __atomic builtins +AC_MSG_CHECKING(for GCC >= 4.7 __atomic builtins) +AC_LINK_IFELSE( +[ + AC_LANG_SOURCE([[ + volatile int val = 1; + int main() { + __atomic_load_n(&val, __ATOMIC_SEQ_CST); + return 0; + } + ]]) +],[have_builtin_atomic=yes],[have_builtin_atomic=no]) + +AC_MSG_RESULT($have_builtin_atomic) + +if test "$have_builtin_atomic" = yes; then + AC_DEFINE(HAVE_BUILTIN_ATOMIC, 1, [Has builtin atomics]) +fi + # ensurepip option AC_MSG_CHECKING(for ensurepip) AC_ARG_WITH(ensurepip, diff --git a/pyconfig.h.in b/pyconfig.h.in --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -101,6 +101,9 @@ /* Define if `unsetenv` does not return an int. */ #undef HAVE_BROKEN_UNSETENV +/* Has builtin atomics */ +#undef HAVE_BUILTIN_ATOMIC + /* Define this if you have the type _Bool. */ #undef HAVE_C99_BOOL @@ -877,6 +880,9 @@ /* Define to 1 if you have the header file. */ #undef HAVE_STDLIB_H +/* Has stdatomic.h */ +#undef HAVE_STD_ATOMIC + /* Define to 1 if you have the `strdup' function. */ #undef HAVE_STRDUP -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Fri Jan 9 08:53:56 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 09 Jan 2015 08:53:56 +0100 Subject: [Python-checkins] Daily reference leaks (fbe87fb071a6): sum=6 Message-ID: results for fbe87fb071a6 on branch "default" -------------------------------------------- test_asyncio leaked [0, 3, 0] memory blocks, sum=3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogCP4DXp', '-x'] From python-checkins at python.org Fri Jan 9 16:00:50 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 15:00:50 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?= Message-ID: <20150109150037.22405.78604@psf.io> https://hg.python.org/cpython/rev/d428b4ee8b39 changeset: 94090:d428b4ee8b39 parent: 94087:fbe87fb071a6 parent: 94089:75d7f29487ce user: Victor Stinner date: Fri Jan 09 16:00:30 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-eventloop.rst | 10 +++++----- Doc/library/asyncio-eventloops.rst | 3 ++- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -681,12 +681,12 @@ Event loop examples -=================== +------------------- .. _asyncio-hello-world-callback: Hello World with call_soon() ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Example using the :meth:`BaseEventLoop.call_soon` method to schedule a callback. The callback displays ``"Hello World"`` and then stops the event @@ -716,7 +716,7 @@ .. _asyncio-date-callback: Display the current date with call_later() ------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Example of callback displaying the current date every second. The callback uses the :meth:`BaseEventLoop.call_later` method to reschedule itself during 5 @@ -752,7 +752,7 @@ .. _asyncio-watch-read-event: Watch a file descriptor for read events ---------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Wait until a file descriptor received some data using the :meth:`BaseEventLoop.add_reader` method and then close the event loop:: @@ -801,7 +801,7 @@ Set signal handlers for SIGINT and SIGTERM ------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Register handlers for signals :py:data:`SIGINT` and :py:data:`SIGTERM` using the :meth:`BaseEventLoop.add_signal_handler` method:: diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst --- a/Doc/library/asyncio-eventloops.rst +++ b/Doc/library/asyncio-eventloops.rst @@ -87,7 +87,8 @@ :class:`SelectorEventLoop` specific limits: -- :class:`~selectors.SelectSelector` is used but it only supports sockets +- :class:`~selectors.SelectSelector` is used which only supports sockets + and is limited to 512 sockets. - :meth:`~BaseEventLoop.add_reader` and :meth:`~BaseEventLoop.add_writer` only accept file descriptors of sockets - Pipes are not supported -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 16:00:50 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 15:00:50 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogU2Vs?= =?utf-8?q?ectSelector_is_limited_to_512_sockets_on_Windows?= Message-ID: <20150109150037.22423.65117@psf.io> https://hg.python.org/cpython/rev/75d7f29487ce changeset: 94089:75d7f29487ce branch: 3.4 user: Victor Stinner date: Fri Jan 09 15:59:44 2015 +0100 summary: asyncio: SelectSelector is limited to 512 sockets on Windows files: Doc/library/asyncio-eventloops.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst --- a/Doc/library/asyncio-eventloops.rst +++ b/Doc/library/asyncio-eventloops.rst @@ -87,7 +87,8 @@ :class:`SelectorEventLoop` specific limits: -- :class:`~selectors.SelectSelector` is used but it only supports sockets +- :class:`~selectors.SelectSelector` is used which only supports sockets + and is limited to 512 sockets. - :meth:`~BaseEventLoop.add_reader` and :meth:`~BaseEventLoop.add_writer` only accept file descriptors of sockets - Pipes are not supported -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 16:00:50 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 15:00:50 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbyBkb2M6?= =?utf-8?q?_fix_section_of_event_loop_examples?= Message-ID: <20150109150036.125890.96457@psf.io> https://hg.python.org/cpython/rev/f2b9fac7d040 changeset: 94088:f2b9fac7d040 branch: 3.4 parent: 94085:d07386812daa user: Victor Stinner date: Fri Jan 09 15:58:41 2015 +0100 summary: asyncio doc: fix section of event loop examples files: Doc/library/asyncio-eventloop.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -681,12 +681,12 @@ Event loop examples -=================== +------------------- .. _asyncio-hello-world-callback: Hello World with call_soon() ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Example using the :meth:`BaseEventLoop.call_soon` method to schedule a callback. The callback displays ``"Hello World"`` and then stops the event @@ -716,7 +716,7 @@ .. _asyncio-date-callback: Display the current date with call_later() ------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Example of callback displaying the current date every second. The callback uses the :meth:`BaseEventLoop.call_later` method to reschedule itself during 5 @@ -752,7 +752,7 @@ .. _asyncio-watch-read-event: Watch a file descriptor for read events ---------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Wait until a file descriptor received some data using the :meth:`BaseEventLoop.add_reader` method and then close the event loop:: @@ -801,7 +801,7 @@ Set signal handlers for SIGINT and SIGTERM ------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Register handlers for signals :py:data:`SIGINT` and :py:data:`SIGTERM` using the :meth:`BaseEventLoop.add_signal_handler` method:: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 17:39:41 2015 From: python-checkins at python.org (brett.cannon) Date: Fri, 09 Jan 2015 16:39:41 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323014=3A_Make_imp?= =?utf-8?q?ortlib=2Eabc=2ELoader=2Ecreate=5Fmodule=28=29_required_when?= Message-ID: <20150109163931.125896.6214@psf.io> https://hg.python.org/cpython/rev/ab72f30bcd9f changeset: 94091:ab72f30bcd9f user: Brett Cannon date: Fri Jan 09 11:39:21 2015 -0500 summary: Issue #23014: Make importlib.abc.Loader.create_module() required when importlib.abc.Loader.exec_module() is also defined. Before this change, create_module() was optional **and** could return None to trigger default semantics. This change now reduces the options for choosing default semantics to one and in the most backporting-friendly way (define create_module() to return None). files: Doc/library/importlib.rst | 24 +- Doc/reference/import.rst | 8 +- Doc/whatsnew/3.5.rst | 8 + Lib/importlib/_bootstrap.py | 14 + Lib/importlib/abc.py | 3 - Lib/test/test_importlib/import_/test___loader__.py | 3 + Lib/test/test_importlib/import_/test_api.py | 4 + Lib/test/test_importlib/test_spec.py | 3 + Lib/test/test_importlib/test_util.py | 12 +- Lib/test/test_pkgutil.py | 3 + Python/importlib.h | 4719 +++++---- 11 files changed, 2447 insertions(+), 2354 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -347,13 +347,16 @@ .. method:: create_module(spec) - An optional method that returns the module object to use when - importing a module. create_module() may also return ``None``, - indicating that the default module creation should take place - instead. + A method that returns the module object to use when + importing a module. This method may return ``None``, + indicating that default module creation semantics should take place. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Starting in Python 3.6, this method will not be optional when + :meth:`exec_module` is defined. + .. method:: exec_module(module) An abstract method that executes the module in its own namespace @@ -417,7 +420,7 @@ .. deprecated:: 3.4 The recommended API for loading a module is :meth:`exec_module` - (and optionally :meth:`create_module`). Loaders should implement + (and :meth:`create_module`). Loaders should implement it instead of load_module(). The import machinery takes care of all the other responsibilities of load_module() when exec_module() is implemented. @@ -1136,9 +1139,9 @@ .. function:: module_from_spec(spec) - Create a new module based on **spec**. + Create a new module based on **spec** and ``spec.loader.create_module()``. - If the module object is from ``spec.loader.create_module()``, then any + If ``spec.loader.create_module()`` does not return ``None``, then any pre-existing attributes will not be reset. Also, no :exc:`AttributeError` will be raised if triggered while accessing **spec** or setting an attribute on the module. @@ -1234,9 +1237,10 @@ module has an attribute accessed. This class **only** works with loaders that define - :meth:`importlib.abc.Loader.exec_module` as control over what module type - is used for the module is required. For the same reasons, the loader - **cannot** define :meth:`importlib.abc.Loader.create_module`. Finally, + :meth:`~importlib.abc.Loader.exec_module` as control over what module type + is used for the module is required. For those same reasons, the loader's + :meth:`~importlib.abc.Loader.create_module` method will be ignored (i.e., the + loader's method should only return ``None``). Finally, modules which substitute the object placed into :attr:`sys.modules` will not work as there is no way to properly replace the module references throughout the interpreter safely; :exc:`ValueError` is raised if such a diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -339,6 +339,7 @@ module = None if spec.loader is not None and hasattr(spec.loader, 'create_module'): + # It is assumed 'exec_module' will also be defined on the loader. module = spec.loader.create_module(spec) if module is None: module = ModuleType(spec.name) @@ -427,7 +428,7 @@ by implementing a :meth:`~importlib.abc.Loader.create_module` method. It takes one argument, the module spec, and returns the new module object to use during loading. ``create_module()`` does not need to set any attributes -on the module object. If the loader does not define ``create_module()``, the +on the module object. If the method returns ``None``, the import machinery will create the new module itself. .. versionadded:: 3.4 @@ -462,6 +463,11 @@ module(s), and only if the loader itself has loaded the module(s) explicitly. +.. versionchanged:: 3.5 + A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but + ``create_module()`` is not. Starting in Python 3.6 it will be an error to not + define ``create_module()`` on a loader attached to a ModuleSpec. + Module spec ----------- diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -456,6 +456,14 @@ `http.client` and `http.server` remain available for backwards compatibility. (Contributed by Demian Brecht in :issue:`21793`.) +* When an import loader defines :meth:`~importlib.machinery.Loader.exec_module` + it is now expected to also define + :meth:`~importlib.machinery.Loader.create_module` (raises a + :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader + inherits from :class:`importlib.abc.Loader` then there is nothing to do, else + simply define :meth:`~importlib.machinery.Loader.create_module` to return + ``None`` (:issue:`23014`). + Changes in the C API -------------------- diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1055,6 +1055,10 @@ # If create_module() returns `None` then it means default # module creation should be used. module = spec.loader.create_module(spec) + elif hasattr(spec.loader, 'exec_module'): + _warnings.warn('starting in Python 3.6, loaders defining exec_module() ' + 'must also define create_module()', + DeprecationWarning, stacklevel=2) if module is None: module = _new_module(spec.name) _init_module_attrs(spec, module) @@ -1298,6 +1302,10 @@ """ return cls if _imp.is_frozen(fullname) else None + @classmethod + def create_module(cls, spec): + """Use default semantics for module creation.""" + @staticmethod def exec_module(module): name = module.__spec__.name @@ -1411,6 +1419,9 @@ tail_name = fullname.rpartition('.')[2] return filename_base == '__init__' and tail_name != '__init__' + def create_module(self, spec): + """Use default semantics for module creation.""" + def exec_module(self, module): """Execute the module.""" code = self.get_code(module.__name__) @@ -1771,6 +1782,9 @@ def get_code(self, fullname): return compile('', '', 'exec', dont_inherit=True) + def create_module(self, spec): + """Use default semantics for module creation.""" + def exec_module(self, module): pass diff --git a/Lib/importlib/abc.py b/Lib/importlib/abc.py --- a/Lib/importlib/abc.py +++ b/Lib/importlib/abc.py @@ -122,9 +122,6 @@ This method should raise ImportError if anything prevents it from creating a new module. It may return None to indicate that the spec should create the new module. - - create_module() is optional. - """ # By default, defer to default semantics for the new module. return None diff --git a/Lib/test/test_importlib/import_/test___loader__.py b/Lib/test/test_importlib/import_/test___loader__.py --- a/Lib/test/test_importlib/import_/test___loader__.py +++ b/Lib/test/test_importlib/import_/test___loader__.py @@ -11,6 +11,9 @@ def find_spec(self, fullname, path=None, target=None): return machinery.ModuleSpec(fullname, self) + def create_module(self, spec): + return None + def exec_module(self, module): pass diff --git a/Lib/test/test_importlib/import_/test_api.py b/Lib/test/test_importlib/import_/test_api.py --- a/Lib/test/test_importlib/import_/test_api.py +++ b/Lib/test/test_importlib/import_/test_api.py @@ -17,6 +17,10 @@ return spec @staticmethod + def create_module(spec): + return None + + @staticmethod def exec_module(module): if module.__name__ == SUBMOD_NAME: raise ImportError('I cannot be loaded!') diff --git a/Lib/test/test_importlib/test_spec.py b/Lib/test/test_importlib/test_spec.py --- a/Lib/test/test_importlib/test_spec.py +++ b/Lib/test/test_importlib/test_spec.py @@ -34,6 +34,9 @@ def _is_package(self, name): return self.package + def create_module(self, spec): + return None + class NewLoader(TestLoader): diff --git a/Lib/test/test_importlib/test_util.py b/Lib/test/test_importlib/test_util.py --- a/Lib/test/test_importlib/test_util.py +++ b/Lib/test/test_importlib/test_util.py @@ -41,10 +41,16 @@ class ModuleFromSpecTests: def test_no_create_module(self): - class Loader(self.abc.Loader): - pass + class Loader: + def exec_module(self, module): + pass spec = self.machinery.ModuleSpec('test', Loader()) - module = self.util.module_from_spec(spec) + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + module = self.util.module_from_spec(spec) + self.assertEqual(1, len(w)) + self.assertTrue(issubclass(w[0].category, DeprecationWarning)) + self.assertIn('create_module', str(w[0].message)) self.assertIsInstance(module, types.ModuleType) self.assertEqual(module.__name__, spec.name) diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -104,6 +104,9 @@ class PkgutilPEP302Tests(unittest.TestCase): class MyTestLoader(object): + def create_module(self, spec): + return None + def exec_module(self, mod): # Count how many times the module is reloaded mod.__dict__['loads'] = mod.__dict__.get('loads', 0) + 1 diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 19:19:40 2015 From: python-checkins at python.org (chris.angelico) Date: Fri, 09 Jan 2015 18:19:40 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Import_Guido=27s_text_from_ht?= =?utf-8?q?tps=3A//quip=2Ecom/r69HA9GhGa7J_into_PEP_483?= Message-ID: <20150109181934.125900.72027@psf.io> https://hg.python.org/peps/rev/ed50a618f8ef changeset: 5663:ed50a618f8ef user: Chris Angelico date: Sat Jan 10 05:19:10 2015 +1100 summary: Import Guido's text from https://quip.com/r69HA9GhGa7J into PEP 483 files: pep-0483.txt | 315 ++++++++++++++++++++++++++++++++++++++- 1 files changed, 311 insertions(+), 4 deletions(-) diff --git a/pep-0483.txt b/pep-0483.txt --- a/pep-0483.txt +++ b/pep-0483.txt @@ -11,11 +11,318 @@ Post-History: Resolution: -Abstract -======== +The Theory of Type Hinting +========================== -This PEP is currently a stub. The content should be copied from -https://quip.com/r69HA9GhGa7J and reformatted. +Guido van Rossum, Dec 19, 2014 (with a few more recent updates) + +This work is licensed under a `Creative Commons +Attribution-NonCommercial-ShareAlike 4.0 International +License `_. + + +Introduction +------------ + +This document lays out the theory of the new type hinting proposal for +Python 3.5. It's not quite a full proposal or specification because +there are many details that need to be worked out, but it lays out the +theory without which it is hard to discuss more detailed specifications. +We start by explaining gradual typing; then we state some conventions +and general rules; then we define the new special types (such as Union) +that can be used in annotations; and finally we define the approach to +generic types. (The latter section needs more fleshing out; sorry!) + + +Summary of gradual typing +------------------------- + +We define a new relationship, is-consistent-with, which is similar to +is-subclass-of, except it is not transitive when the new type **Any** is +involved. (Neither relationship is symmetric.) Assigning x to y is OK if +the type of x is consistent with the type of y. (Compare this to??... if +the type of x is a subclass of the type of y,? which states one of the +fundamentals of OO programming.) The is-consistent-with relationship is +defined by three rules: + +- A type t1 is consistent with a type t2 if t1 is a subclass of t2. + (But not the other way around.) +- **Any** is consistent with every type. (But **Any** is not a subclass + of every type.) +- Every type is a subclass of **Any**. (Which also makes every type + consistent with **Any**, via rule 1.) + +That's all! See Jeremy Siek's blog post `What is Gradual +Typing `_ +for a longer explanation and motivation. Note that rule 3 places **Any** +at the root of the class graph. This makes it very similar to +**object**. The difference is that **object** is not consistent with +most types (e.g. you can't use an object() instance where an int is +expected). IOW both **Any** and **object** mean??any type is allowed? +when used to annotate an argument, but only **Any** can be passed no +matter what type is expected (in essence, **Any** shuts up complaints +from the static checker). + +?Here's an example showing how these rules work out in practice: + +Say we have an Employee class, and a subclass Manager: + +- class Employee: ... +- class Manager(Employee): ... + +Let's say variable e is declared with type Employee: + +- e = Employee()? # type: Employee + +Now it's okay to assign a Manager instance to e (rule 1): + +- e = Manager() + +It's not okay to assign an Employee instance to a variable declared with +type Manager: + +- m = Manager()? # type: Manager +- m = Employee()? # Fails static check + +However, suppose we have a variable whose type is **Any**: + +- a = some\_func()? # type: Any + +Now it's okay to assign a to e (rule 2): + +- e = a? # OK + +Of course it's also okay to assign e to a (rule 3), but we didn't need +the concept of consistency for that: + +- a = e? # OK + + +Notational conventions +---------------------- + +- t1, t2 etc.? and u1, u2 etc. are types or classes. Sometimes we write + ti or tj to refer to??any of t1, t2, etc.? +- X, Y etc. are type variables (defined with Var(), see below). +- C, D etc. are classes defined with a class statement. +- x, y etc. are objects or instances. +- We use the terms?type and?class interchangeably, and we assume + type(x) is x.\_\_class\_\_. + + +General rules +------------- + +- Instance-ness is? derived from class-ness, e.g. x is an instance of + t1 if? type(x) is a subclass of t1. +- No types defined below (i.e. Any, Union etc.) can be instantiated. + (But non-abstract subclasses of Generic can be.) +- No types defined below can be subclassed, except for Generic and + classes derived from it. +- Where a type is expected, None can be substituted for type(None); + e.g. Union[t1, None] == Union[t1, type(None)]. + + +Types +----- + +- **Any**. Every class is a subclass of Any; however, to the static + type checker it is also consistent with every class (see above). +- **Union[t1, t2, ...]**. Classes that are subclass of at least one of + t1 etc. are subclasses of this. So are unions whose components are + all subclasses of t1 etc. (Example: Union[int, str] is a subclass of + Union[int, float, str].) The order of the arguments doesn't matter. + (Example: Union[int, str] == Union[str, int].) If ti is itself a + Union the result is flattened. (Example: Union[int, Union[float, + str]] == Union[int, float, str].) If ti and tj have a subclass + relationship, the less specific type survives. (Example: + Union[Employee, Manager] == Union[Employee].)?Union[t1] returns just + t1. Union[] is illegal, so is Union[()]. Corollary: Union[..., Any, + ...] returns Any; Union[..., object, ...] returns object; to cut a + tie, Union[Any, object] == Union[object, Any] == Any. +- **Optional[t1]**. Alias for Union[t1, None], i.e. Union[t1, + type(None)]. +- **Tuple[t1, t2, ..., tn]**. A tuple whose items are instances of t1 + etc.. Example: Tuple[int, float] means a tuple of two items, the + first is an int, the second a float; e.g., (42, 3.14). Tuple[u1, u2, + ..., um] is a subclass of Tuple[t1, t2, ..., tn] if they have the + same length (n==m) and each ui is a subclass of ti. To spell the type + of the empty tuple, use Tuple[()]. There is no way to define a + variadic tuple type. (TODO: Maybe Tuple[t1, ...] with literal + ellipsis?) +- **Callable[[t1, t2, ..., tn], tr]**. A function with positional + argument types t1 etc., and return type tr. The argument list may be + empty (n==0). There is no way to indicate optional or keyword + arguments, nor varargs (we don't need to spell those often enough to + complicate the syntax ? however, Reticulated Python has a useful idea + here). This is covariant in the return type, but contravariant in the + arguments. ?Covariant? here means that for two callable types that + differ only in the return type, the subclass relationship for the + callable types follows that of the return types. (Example: + Callable[[], Manager] is a subclass of Callable[[], Employee].) + ?Contravariant? here means that for two callable types that differ + only in the type of one argument, the subclass relationship for the + callable types goes in the opposite direction as for the argument + types. (Example: Callable[[Employee], None] is a subclass of + Callable[[Mananger], None]. Yes, you read that right.) + +We might add: + +- **Intersection[t1, t2, ...]**. Classes that are subclass of *each* of + t1, etc are subclasses of this. (Compare to Union, which has *at + least one* instead of *each* in its definition.) The order of the + arguments doesn't matter. Nested intersections are flattened, e.g. + Intersection[int, Intersection[float, str]] == Intersection[int, + float, str]. An intersection of fewer types is a subclass of an + intersection of more types, e.g. Intersection[int, str] is a subclass + of Intersection[int, float, str]. An intersection of one argument is + just that argument, e.g. Intersection[int] is int. When argument have + a subclass relationship, the more specific class survives, e.g. + Intersection[str, Employee, Manager] is Intersection[str, Manager]. + Intersection[] is illegal, so is Intersection[()]. Corollary: Any + disappears from the argument list, e.g. Intersection[int, str, Any] + == Intersection[int, str]. Intersection[Any, object] is object. The + interaction between Intersection and Union is complex but should be + no surprise if you understand the interaction between intersections + and unions in set theory (note that sets of types can be infinite in + size, since there is no limit on the number of new subclasses). + + +Pragmatics +---------- + +Some things are irrelevant to the theory but make practical use more +convenient. (This is not a full list; I probably missed a few and some +are still controversial or not fully specified.) + +- Type aliases, e.g. + + * point = Tuple[float, float] + * def distance(p: point) -> float: ...? + +- Forward references via strings, e.g. + + * class C: + + + def compare(self, other: ?C?) -> int: ... + +- If a default of None is specified, the type is implicitly optional, e.g. + + * def get(key: KT, default: VT = None) -> VT: ... + +- Don't use dynamic type expressions; use builtins and imported types + only. No 'if'. + + * def display(message: str if WINDOWS else bytes):? # NOT OK + +- Type declaration in comments, e.g. + + * x = []? # type: Sequence[int] + +- Type declarations using Undefined, e.g. + + * x = Undefined(str) + +- Other things, e.g. casts, overloading and stub modules; best left to an + actual PEP. + + +Generic types +------------- + +(TODO: Explain more. See also the `mypy docs on +generics `_.) + +- **X = Var('X')**. Declares a unique type variable. The name must match + the variable name. + +- **Y = Var('Y', t1, t2, ...).** Ditto, constrained to t1 etc. Behaves + ?like Union[t1, t2, ...] for most purposes, but when used as a type + variable, subclasses of t1 etc. are replaced by the most-derived base + class among t1 etc. + +- Example of constrained type variables: + + * AnyStr = Var('AnyStr', str, bytes) + + * def longest(a: AnyStr, b: AnyStr) -> AnyStr: + + - return a if len(a) >= len(b) else b + + * x = longest('a', 'abc')? # The inferred type for x is str + + * y = longest('a', b'abc')? # Fails static type check + + * In this example, both arguments to longest() must have the same type + (str or bytes), and moreover, even if the arguments are instances of a + common str subclass, the return type is still str, not that subclass + (see next example). + +- For comparison, if the type variable was unconstrained, the common + subclass would be chosen as the return type, e.g.: + + * S = Var('S') + + * def longest(a: S, b: S) -> S: + + - return a if len(a) >= b else b + + * class MyStr(str): ... + + * x = longest(MyStr('a'), MyStr('abc')) + + * The inferred type of x is MyStr (whereas in the AnyStr example it would + be str). + +- Also for comparison, if a Union is used, the return type also has to be + a Union: + + * U = Union[str, bytes] + + * def longest(a: U, b: U) -> U: + + - return a if len(a) >= b else b + + * x = longest('a', 'abc') + + * The inferred type of x is still Union[str, bytes], even though both + arguments are str. + +- **class C(Generic[X, Y, ...]):** ... Define a generic class C over type + variables X etc. C itself becomes parameterizable, e.g. C[int, str, ...] + is a specific class with substitutions X?int etc. + +- TODO: Explain use of generic types in function signatures. E.g. + Sequence[X], Sequence[int], Sequence[Tuple[X, Y, Z]], and mixtures. + Think about co\*variance. No gimmicks like deriving from + Sequence[Union[int, str]] or Sequence[Union[int, X]]. + +- **Protocol**. Similar to Generic but uses structural equivalence. (TODO: + Explain, and think about co\*variance.) + + +Predefined generic types and Protocols in typing.py +--------------------------------------------------- + +(See also the `mypy typing.py +module `_.) + +- Everything from collections.abc (but Set renamed to AbstractSet). +- Dict, List, Set, a few more. (FrozenSet?) +- Pattern, Match. (Why?) +- IO, TextIO, BinaryIO. (Why?) + + +Another reference +----------------- + +Lest mypy gets all the attention, I should mention?\ `Reticulated +Python `_ by Michael Vitousek +as an example of a slightly different approach to gradual typing for +Python. It is described in an actual `academic +paper `_ +written by Vitousek with Jeremy Siek and Jim Baker (the latter of Jython +fame). .. -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 9 19:29:54 2015 From: python-checkins at python.org (chris.angelico) Date: Fri, 09 Jan 2015 18:29:54 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_ASCIIfy_PEP_483?= Message-ID: <20150109182942.11571.43952@psf.io> https://hg.python.org/peps/rev/003429214c06 changeset: 5664:003429214c06 user: Chris Angelico date: Sat Jan 10 05:28:21 2015 +1100 summary: ASCIIfy PEP 483 files: pep-0483.txt | 56 ++++++++++++++++++++-------------------- 1 files changed, 28 insertions(+), 28 deletions(-) diff --git a/pep-0483.txt b/pep-0483.txt --- a/pep-0483.txt +++ b/pep-0483.txt @@ -40,8 +40,8 @@ We define a new relationship, is-consistent-with, which is similar to is-subclass-of, except it is not transitive when the new type **Any** is involved. (Neither relationship is symmetric.) Assigning x to y is OK if -the type of x is consistent with the type of y. (Compare this to??... if -the type of x is a subclass of the type of y,? which states one of the +the type of x is consistent with the type of y. (Compare this to "... if +the type of x is a subclass of the type of y," which states one of the fundamentals of OO programming.) The is-consistent-with relationship is defined by three rules: @@ -58,12 +58,12 @@ at the root of the class graph. This makes it very similar to **object**. The difference is that **object** is not consistent with most types (e.g. you can't use an object() instance where an int is -expected). IOW both **Any** and **object** mean??any type is allowed? +expected). IOW both **Any** and **object** mean "any type is allowed" when used to annotate an argument, but only **Any** can be passed no matter what type is expected (in essence, **Any** shuts up complaints from the static checker). -?Here's an example showing how these rules work out in practice: +Here's an example showing how these rules work out in practice: Say we have an Employee class, and a subclass Manager: @@ -72,7 +72,7 @@ Let's say variable e is declared with type Employee: -- e = Employee()? # type: Employee +- e = Employee() # type: Employee Now it's okay to assign a Manager instance to e (rule 1): @@ -81,40 +81,40 @@ It's not okay to assign an Employee instance to a variable declared with type Manager: -- m = Manager()? # type: Manager -- m = Employee()? # Fails static check +- m = Manager() # type: Manager +- m = Employee() # Fails static check However, suppose we have a variable whose type is **Any**: -- a = some\_func()? # type: Any +- a = some\_func() # type: Any Now it's okay to assign a to e (rule 2): -- e = a? # OK +- e = a # OK Of course it's also okay to assign e to a (rule 3), but we didn't need the concept of consistency for that: -- a = e? # OK +- a = e # OK Notational conventions ---------------------- -- t1, t2 etc.? and u1, u2 etc. are types or classes. Sometimes we write - ti or tj to refer to??any of t1, t2, etc.? +- t1, t2 etc. and u1, u2 etc. are types or classes. Sometimes we write + ti or tj to refer to "any of t1, t2, etc." - X, Y etc. are type variables (defined with Var(), see below). - C, D etc. are classes defined with a class statement. - x, y etc. are objects or instances. -- We use the terms?type and?class interchangeably, and we assume +- We use the terms type and class interchangeably, and we assume type(x) is x.\_\_class\_\_. General rules ------------- -- Instance-ness is? derived from class-ness, e.g. x is an instance of - t1 if? type(x) is a subclass of t1. +- Instance-ness is derived from class-ness, e.g. x is an instance of + t1 if type(x) is a subclass of t1. - No types defined below (i.e. Any, Union etc.) can be instantiated. (But non-abstract subclasses of Generic can be.) - No types defined below can be subclassed, except for Generic and @@ -136,7 +136,7 @@ Union the result is flattened. (Example: Union[int, Union[float, str]] == Union[int, float, str].) If ti and tj have a subclass relationship, the less specific type survives. (Example: - Union[Employee, Manager] == Union[Employee].)?Union[t1] returns just + Union[Employee, Manager] == Union[Employee].) Union[t1] returns just t1. Union[] is illegal, so is Union[()]. Corollary: Union[..., Any, ...] returns Any; Union[..., object, ...] returns object; to cut a tie, Union[Any, object] == Union[object, Any] == Any. @@ -154,13 +154,13 @@ argument types t1 etc., and return type tr. The argument list may be empty (n==0). There is no way to indicate optional or keyword arguments, nor varargs (we don't need to spell those often enough to - complicate the syntax ? however, Reticulated Python has a useful idea + complicate the syntax - however, Reticulated Python has a useful idea here). This is covariant in the return type, but contravariant in the - arguments. ?Covariant? here means that for two callable types that + arguments. "Covariant" here means that for two callable types that differ only in the return type, the subclass relationship for the callable types follows that of the return types. (Example: Callable[[], Manager] is a subclass of Callable[[], Employee].) - ?Contravariant? here means that for two callable types that differ + "Contravariant" here means that for two callable types that differ only in the type of one argument, the subclass relationship for the callable types goes in the opposite direction as for the argument types. (Example: Callable[[Employee], None] is a subclass of @@ -198,13 +198,13 @@ - Type aliases, e.g. * point = Tuple[float, float] - * def distance(p: point) -> float: ...? + * def distance(p: point) -> float: ... - Forward references via strings, e.g. * class C: - + def compare(self, other: ?C?) -> int: ... + + def compare(self, other: "C") -> int: ... - If a default of None is specified, the type is implicitly optional, e.g. @@ -213,11 +213,11 @@ - Don't use dynamic type expressions; use builtins and imported types only. No 'if'. - * def display(message: str if WINDOWS else bytes):? # NOT OK + * def display(message: str if WINDOWS else bytes): # NOT OK - Type declaration in comments, e.g. - * x = []? # type: Sequence[int] + * x = [] # type: Sequence[int] - Type declarations using Undefined, e.g. @@ -237,7 +237,7 @@ the variable name. - **Y = Var('Y', t1, t2, ...).** Ditto, constrained to t1 etc. Behaves - ?like Union[t1, t2, ...] for most purposes, but when used as a type + like Union[t1, t2, ...] for most purposes, but when used as a type variable, subclasses of t1 etc. are replaced by the most-derived base class among t1 etc. @@ -249,9 +249,9 @@ - return a if len(a) >= len(b) else b - * x = longest('a', 'abc')? # The inferred type for x is str + * x = longest('a', 'abc') # The inferred type for x is str - * y = longest('a', b'abc')? # Fails static type check + * y = longest('a', b'abc') # Fails static type check * In this example, both arguments to longest() must have the same type (str or bytes), and moreover, even if the arguments are instances of a @@ -290,7 +290,7 @@ - **class C(Generic[X, Y, ...]):** ... Define a generic class C over type variables X etc. C itself becomes parameterizable, e.g. C[int, str, ...] - is a specific class with substitutions X?int etc. + is a specific class with substitutions X->int etc. - TODO: Explain use of generic types in function signatures. E.g. Sequence[X], Sequence[int], Sequence[Tuple[X, Y, Z]], and mixtures. @@ -316,7 +316,7 @@ Another reference ----------------- -Lest mypy gets all the attention, I should mention?\ `Reticulated +Lest mypy gets all the attention, I should mention \ `Reticulated Python `_ by Michael Vitousek as an example of a slightly different approach to gradual typing for Python. It is described in an actual `academic -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 9 19:40:04 2015 From: python-checkins at python.org (chris.angelico) Date: Fri, 09 Jan 2015 18:40:04 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Record_Guido=27s_relicensing_?= =?utf-8?q?of_PEP_483_under_Open_Publication_License?= Message-ID: <20150109184000.125886.5154@psf.io> https://hg.python.org/peps/rev/1cdccde25189 changeset: 5665:1cdccde25189 user: Chris Angelico date: Sat Jan 10 05:39:44 2015 +1100 summary: Record Guido's relicensing of PEP 483 under Open Publication License files: pep-0483.txt | 20 +++++++++++++------- 1 files changed, 13 insertions(+), 7 deletions(-) diff --git a/pep-0483.txt b/pep-0483.txt --- a/pep-0483.txt +++ b/pep-0483.txt @@ -14,13 +14,6 @@ The Theory of Type Hinting ========================== -Guido van Rossum, Dec 19, 2014 (with a few more recent updates) - -This work is licensed under a `Creative Commons -Attribution-NonCommercial-ShareAlike 4.0 International -License `_. - - Introduction ------------ @@ -324,6 +317,19 @@ written by Vitousek with Jeremy Siek and Jim Baker (the latter of Jython fame). + +Copyright +========= + +This document is licensed under the `Open Publication License`_. + + +References and Footnotes +======================== + +.. _Open Publication License: http://www.opencontent.org/openpub/ + + .. Local Variables: -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 9 19:55:35 2015 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 09 Jan 2015 18:55:35 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Add_brief_section_on_Reticula?= =?utf-8?q?ted_Python=2C_plus_stubs_for_other_Pythonic_projects=2E?= Message-ID: <20150109185515.11591.5159@psf.io> https://hg.python.org/peps/rev/39fedd54da48 changeset: 5666:39fedd54da48 user: Guido van Rossum date: Fri Jan 09 10:52:06 2015 -0800 summary: Add brief section on Reticulated Python, plus stubs for other Pythonic projects. files: pep-0482.txt | 35 ++++++++++++++++++++++++++++++----- 1 files changed, 30 insertions(+), 5 deletions(-) diff --git a/pep-0482.txt b/pep-0482.txt --- a/pep-0482.txt +++ b/pep-0482.txt @@ -18,8 +18,9 @@ literature overview of related work. -Existing Approaches in Other Languages -====================================== +Existing Approaches for Python +============================== + mypy ---- @@ -27,6 +28,27 @@ (This section is a stub, since mypy [mypy]_ is essentially what we're proposing.) + +Reticulated Python +------------------ + +Reticulated Python [reticulated]_ by Michael Vitousek is an example of +a slightly different approach to gradual typing for Python. It is +described in an actual academic paper [reticulated-paper]_ written by +Vitousek with Jeremy Siek and Jim Baker (the latter of Jython fame). + + +Others +------ + +TBD: Add sections on pyflakes [pyflakes]_, pylint [pylint]_, numpy +[numpy]_, Argument Clinic [argumentclinic]_, pytypedecl [pytypedecl]_, +numba [numba]_, obiwan [obiwan]_. + + +Existing Approaches in Other Languages +====================================== + ActionScript ------------ @@ -165,12 +187,15 @@ References ========== -.. [pep-3107] - http://www.python.org/dev/peps/pep-3107/ - .. [mypy] http://mypy-lang.org +.. [reticulated] + https://github.com/mvitousek/reticulated + +.. [reticulated-paper] + http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf + .. [obiwan] http://pypi.python.org/pypi/obiwan -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 9 19:55:35 2015 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 09 Jan 2015 18:55:35 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Change_the_level_of_a_few_hea?= =?utf-8?q?dings=2E_Remove_duplicate_title=2E?= Message-ID: <20150109185515.8741.71799@psf.io> https://hg.python.org/peps/rev/ac3c4fefd85f changeset: 5668:ac3c4fefd85f user: Guido van Rossum date: Fri Jan 09 10:55:11 2015 -0800 summary: Change the level of a few headings. Remove duplicate title. files: pep-0483.txt | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/pep-0483.txt b/pep-0483.txt --- a/pep-0483.txt +++ b/pep-0483.txt @@ -11,17 +11,14 @@ Post-History: Resolution: -The Theory of Type Hinting -========================== - Abstract --------- +======== This PEP lays out the theory to be referenced by PEP 484. Introduction ------------- +============ This document lays out the theory of the new type hinting proposal for Python 3.5. It's not quite a full proposal or specification because @@ -33,6 +30,10 @@ generic types. (The latter section needs more fleshing out; sorry!) +Specification +============= + + Summary of gradual typing ------------------------- -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 9 19:55:35 2015 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 09 Jan 2015 18:55:35 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Move_text_on_Reticulated_Pyth?= =?utf-8?q?on_to_PEP_483=2E__Add_an_abstract=2E__Set_creation?= Message-ID: <20150109185515.125900.78716@psf.io> https://hg.python.org/peps/rev/67ccdc3a0390 changeset: 5667:67ccdc3a0390 user: Guido van Rossum date: Fri Jan 09 10:53:06 2015 -0800 summary: Move text on Reticulated Python to PEP 483. Add an abstract. Set creation date to 19-Dec-2014 (cretion date of the original Quip doc). files: pep-0483.txt | 20 +++++++------------- 1 files changed, 7 insertions(+), 13 deletions(-) diff --git a/pep-0483.txt b/pep-0483.txt --- a/pep-0483.txt +++ b/pep-0483.txt @@ -7,13 +7,19 @@ Status: Draft Type: Informational Content-Type: text/x-rst -Created: 08-Jan-2015 +Created: 19-Dec-2014 Post-History: Resolution: The Theory of Type Hinting ========================== +Abstract +-------- + +This PEP lays out the theory to be referenced by PEP 484. + + Introduction ------------ @@ -306,18 +312,6 @@ - IO, TextIO, BinaryIO. (Why?) -Another reference ------------------ - -Lest mypy gets all the attention, I should mention \ `Reticulated -Python `_ by Michael Vitousek -as an example of a slightly different approach to gradual typing for -Python. It is described in an actual `academic -paper `_ -written by Vitousek with Jeremy Siek and Jim Baker (the latter of Jython -fame). - - Copyright ========= -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 9 21:36:37 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 20:36:37 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjA5?= =?utf-8?q?=3A_Break_some_reference_cycles_in_asyncio=2E_Patch_written_by_?= =?utf-8?q?Martin?= Message-ID: <20150109203626.8745.28461@psf.io> https://hg.python.org/cpython/rev/376c5398f28d changeset: 94094:376c5398f28d branch: 3.4 parent: 94092:b6a636823c8c user: Victor Stinner date: Fri Jan 09 21:34:27 2015 +0100 summary: Issue #23209: Break some reference cycles in asyncio. Patch written by Martin Richard. files: Lib/asyncio/base_subprocess.py | 1 + Lib/asyncio/futures.py | 2 +- Lib/selectors.py | 1 + 3 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -182,6 +182,7 @@ def connection_lost(self, exc): self.disconnected = True self.proc._pipe_connection_lost(self.fd, exc) + self.proc = None def pause_writing(self): self.proc._protocol.pause_writing() diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -405,5 +405,5 @@ new_future.add_done_callback(_check_cancel_other) fut.add_done_callback( lambda future: loop.call_soon_threadsafe( - new_future._copy_state, fut)) + new_future._copy_state, future)) return new_future diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -256,6 +256,7 @@ def close(self): self._fd_to_key.clear() + self._map = None def get_map(self): return self._map -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 21:36:37 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 20:36:37 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150109203626.22425.7195@psf.io> https://hg.python.org/cpython/rev/242ef936add7 changeset: 94093:242ef936add7 parent: 94091:ab72f30bcd9f parent: 94092:b6a636823c8c user: Victor Stinner date: Fri Jan 09 21:32:24 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/futures.py | 2 - Lib/asyncio/proactor_events.py | 3 +- Lib/asyncio/selector_events.py | 1 - Lib/asyncio/streams.py | 10 +++++-- Lib/asyncio/unix_events.py | 5 +--- Lib/test/test_asyncio/test_streams.py | 19 +++++++++++++++ 6 files changed, 29 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -20,7 +20,6 @@ _PY34 = sys.version_info >= (3, 4) -# TODO: Do we really want to depend on concurrent.futures internals? Error = concurrent.futures._base.Error CancelledError = concurrent.futures.CancelledError TimeoutError = concurrent.futures.TimeoutError @@ -30,7 +29,6 @@ class InvalidStateError(Error): """The operation is not allowed in this state.""" - # TODO: Show the future, its state, the method, and the required state. class _TracebackLogger: diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -487,7 +487,8 @@ self.call_soon(loop) def _process_events(self, event_list): - pass # XXX hard work currently done in poll + # Events are processed in the IocpProactor._poll() method + pass def _stop_accept_futures(self): for future in self._accept_futures.values(): diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -145,7 +145,6 @@ pass # False alarm. except OSError as exc: # There's nowhere to send the error, so just log it. - # TODO: Someone will want an error handler for this. if exc.errno in (errno.EMFILE, errno.ENFILE, errno.ENOBUFS, errno.ENOMEM): # Some platforms (e.g. Linux keep reporting the FD as diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -145,7 +145,10 @@ """ def __init__(self, loop=None): - self._loop = loop # May be None; we may never need it. + if loop is None: + self._loop = events.get_event_loop() + else: + self._loop = loop self._paused = False self._drain_waiter = None self._connection_lost = False @@ -306,8 +309,9 @@ # it also doubles as half the buffer limit. self._limit = limit if loop is None: - loop = events.get_event_loop() - self._loop = loop + self._loop = events.get_event_loop() + else: + self._loop = loop self._buffer = bytearray() self._eof = False # Whether we're done. self._waiter = None # A future. diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -496,9 +496,6 @@ def can_write_eof(self): return True - # TODO: Make the relationships between write_eof(), close(), - # abort(), _fatal_error() and _close() more straightforward. - def write_eof(self): if self._closing: return @@ -897,7 +894,7 @@ class _UnixDefaultEventLoopPolicy(events.BaseDefaultEventLoopPolicy): - """XXX""" + """UNIX event loop policy with a watcher for child processes.""" _loop_factory = _UnixSelectorEventLoop def __init__(self): diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -625,6 +625,25 @@ data = self.loop.run_until_complete(reader.read(-1)) self.assertEqual(data, b'data') + def test_streamreader_constructor(self): + self.addCleanup(asyncio.set_event_loop, None) + asyncio.set_event_loop(self.loop) + + # Tulip issue #184: Ensure that StreamReaderProtocol constructor + # retrieves the current loop if the loop parameter is not set + reader = asyncio.StreamReader() + self.assertIs(reader._loop, self.loop) + + def test_streamreaderprotocol_constructor(self): + self.addCleanup(asyncio.set_event_loop, None) + asyncio.set_event_loop(self.loop) + + # Tulip issue #184: Ensure that StreamReaderProtocol constructor + # retrieves the current loop if the loop parameter is not set + reader = mock.Mock() + protocol = asyncio.StreamReaderProtocol(reader) + self.assertIs(protocol._loop, self.loop) + if __name__ == '__main__': unittest.main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 21:36:38 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 20:36:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40IChhc3luY2lvLCBzZWxlY3RvcnMp?= Message-ID: <20150109203626.11569.71704@psf.io> https://hg.python.org/cpython/rev/880a1febd6f5 changeset: 94095:880a1febd6f5 parent: 94093:242ef936add7 parent: 94094:376c5398f28d user: Victor Stinner date: Fri Jan 09 21:35:03 2015 +0100 summary: Merge 3.4 (asyncio, selectors) files: Lib/asyncio/base_subprocess.py | 1 + Lib/asyncio/futures.py | 2 +- Lib/selectors.py | 1 + 3 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -182,6 +182,7 @@ def connection_lost(self, exc): self.disconnected = True self.proc._pipe_connection_lost(self.fd, exc) + self.proc = None def pause_writing(self): self.proc._protocol.pause_writing() diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -405,5 +405,5 @@ new_future.add_done_callback(_check_cancel_other) fut.add_done_callback( lambda future: loop.call_soon_threadsafe( - new_future._copy_state, fut)) + new_future._copy_state, future)) return new_future diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -256,6 +256,7 @@ def close(self): self._fd_to_key.clear() + self._map = None def get_map(self): return self._map -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 21:36:39 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 20:36:39 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogc3lu?= =?utf-8?q?c_with_Tulip?= Message-ID: <20150109203625.8757.82802@psf.io> https://hg.python.org/cpython/rev/b6a636823c8c changeset: 94092:b6a636823c8c branch: 3.4 parent: 94089:75d7f29487ce user: Victor Stinner date: Fri Jan 09 21:32:05 2015 +0100 summary: asyncio: sync with Tulip * Tulip issue 184: FlowControlMixin constructor now get the event loop if the loop parameter is not set. Add unit tests to ensure that constructor of StreamReader and StreamReaderProtocol classes get the event loop. * Remove outdated TODO/XXX files: Lib/asyncio/futures.py | 2 - Lib/asyncio/proactor_events.py | 3 +- Lib/asyncio/selector_events.py | 1 - Lib/asyncio/streams.py | 10 +++++-- Lib/asyncio/unix_events.py | 5 +--- Lib/test/test_asyncio/test_streams.py | 19 +++++++++++++++ 6 files changed, 29 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -20,7 +20,6 @@ _PY34 = sys.version_info >= (3, 4) -# TODO: Do we really want to depend on concurrent.futures internals? Error = concurrent.futures._base.Error CancelledError = concurrent.futures.CancelledError TimeoutError = concurrent.futures.TimeoutError @@ -30,7 +29,6 @@ class InvalidStateError(Error): """The operation is not allowed in this state.""" - # TODO: Show the future, its state, the method, and the required state. class _TracebackLogger: diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -487,7 +487,8 @@ self.call_soon(loop) def _process_events(self, event_list): - pass # XXX hard work currently done in poll + # Events are processed in the IocpProactor._poll() method + pass def _stop_accept_futures(self): for future in self._accept_futures.values(): diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -145,7 +145,6 @@ pass # False alarm. except OSError as exc: # There's nowhere to send the error, so just log it. - # TODO: Someone will want an error handler for this. if exc.errno in (errno.EMFILE, errno.ENFILE, errno.ENOBUFS, errno.ENOMEM): # Some platforms (e.g. Linux keep reporting the FD as diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -145,7 +145,10 @@ """ def __init__(self, loop=None): - self._loop = loop # May be None; we may never need it. + if loop is None: + self._loop = events.get_event_loop() + else: + self._loop = loop self._paused = False self._drain_waiter = None self._connection_lost = False @@ -306,8 +309,9 @@ # it also doubles as half the buffer limit. self._limit = limit if loop is None: - loop = events.get_event_loop() - self._loop = loop + self._loop = events.get_event_loop() + else: + self._loop = loop self._buffer = bytearray() self._eof = False # Whether we're done. self._waiter = None # A future. diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -496,9 +496,6 @@ def can_write_eof(self): return True - # TODO: Make the relationships between write_eof(), close(), - # abort(), _fatal_error() and _close() more straightforward. - def write_eof(self): if self._closing: return @@ -897,7 +894,7 @@ class _UnixDefaultEventLoopPolicy(events.BaseDefaultEventLoopPolicy): - """XXX""" + """UNIX event loop policy with a watcher for child processes.""" _loop_factory = _UnixSelectorEventLoop def __init__(self): diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -625,6 +625,25 @@ data = self.loop.run_until_complete(reader.read(-1)) self.assertEqual(data, b'data') + def test_streamreader_constructor(self): + self.addCleanup(asyncio.set_event_loop, None) + asyncio.set_event_loop(self.loop) + + # Tulip issue #184: Ensure that StreamReaderProtocol constructor + # retrieves the current loop if the loop parameter is not set + reader = asyncio.StreamReader() + self.assertIs(reader._loop, self.loop) + + def test_streamreaderprotocol_constructor(self): + self.addCleanup(asyncio.set_event_loop, None) + asyncio.set_event_loop(self.loop) + + # Tulip issue #184: Ensure that StreamReaderProtocol constructor + # retrieves the current loop if the loop parameter is not set + reader = mock.Mock() + protocol = asyncio.StreamReaderProtocol(reader) + self.assertIs(protocol._loop, self.loop) + if __name__ == '__main__': unittest.main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 21:58:02 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 20:58:02 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjA5?= =?utf-8?q?=3A_Revert_change_on_selectors=2C_test=5Fselectors_failed=2E?= Message-ID: <20150109205800.11581.70403@psf.io> https://hg.python.org/cpython/rev/7438f2e30908 changeset: 94096:7438f2e30908 branch: 3.4 parent: 94094:376c5398f28d user: Victor Stinner date: Fri Jan 09 21:56:28 2015 +0100 summary: Issue #23209: Revert change on selectors, test_selectors failed. files: Lib/selectors.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -256,7 +256,6 @@ def close(self): self._fd_to_key.clear() - self._map = None def get_map(self): return self._map -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 21:58:01 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 09 Jan 2015 20:58:01 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2323209=3A_Revert_change_on_sel?= =?utf-8?q?ectors=2C_test=5Fselectors_failed=2E?= Message-ID: <20150109205801.22409.4228@psf.io> https://hg.python.org/cpython/rev/27cbc877447b changeset: 94097:27cbc877447b parent: 94095:880a1febd6f5 parent: 94096:7438f2e30908 user: Victor Stinner date: Fri Jan 09 21:57:19 2015 +0100 summary: (Merge 3.4) Issue #23209: Revert change on selectors, test_selectors failed. files: Lib/selectors.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -256,7 +256,6 @@ def close(self): self._fd_to_key.clear() - self._map = None def get_map(self): return self._map -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 22:34:20 2015 From: python-checkins at python.org (ned.deily) Date: Fri, 09 Jan 2015 21:34:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMjEy?= =?utf-8?q?=3A_Update_OS_X_installer_build_OpenSSL_to_1=2E0=2E1k=2E?= Message-ID: <20150109213420.11577.75467@psf.io> https://hg.python.org/cpython/rev/a216f349771b changeset: 94098:a216f349771b branch: 2.7 parent: 94075:1c3f8d044589 user: Ned Deily date: Fri Jan 09 13:26:13 2015 -0800 summary: Issue #23212: Update OS X installer build OpenSSL to 1.0.1k. (currently only used for builds with <= 10.5 deployment targets) files: Mac/BuildScript/build-installer.py | 6 +- Mac/BuildScript/openssl_sdk_makedepend.patch | 16 +++++---- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -237,9 +237,9 @@ result.extend([ dict( - name="OpenSSL 1.0.1j", - url="https://www.openssl.org/source/openssl-1.0.1j.tar.gz", - checksum='f7175c9cd3c39bb1907ac8bba9df8ed3', + name="OpenSSL 1.0.1k", + url="https://www.openssl.org/source/openssl-1.0.1k.tar.gz", + checksum='d4f002bd22a56881340105028842ae1f', patches=[ "openssl_sdk_makedepend.patch", ], diff --git a/Mac/BuildScript/openssl_sdk_makedepend.patch b/Mac/BuildScript/openssl_sdk_makedepend.patch --- a/Mac/BuildScript/openssl_sdk_makedepend.patch +++ b/Mac/BuildScript/openssl_sdk_makedepend.patch @@ -1,13 +1,15 @@ # openssl_sdk_makedepend.patch # -# using openssl 1.0.1j +# using openssl 1.0.1k # # - support building with an OS X SDK # - allow "make depend" to use compilers with names other than "gcc" diff Configure ---- a/Configure Fri Dec 05 01:24:16 2014 -0800 -+++ b/Configure Fri Dec 05 01:52:29 2014 -0800 + +diff -r 99ae439a07f1 Configure +--- a/Configure Fri Jan 09 12:50:43 2015 -0800 ++++ b/Configure Fri Jan 09 12:53:52 2015 -0800 @@ -577,11 +577,11 @@ ##### MacOS X (a.k.a. Rhapsody or Darwin) setup @@ -25,7 +27,7 @@ "debug-darwin-ppc-cc","cc:-DBN_DEBUG -DREF_CHECK -DCONF_DEBUG -DCRYPTO_MDEBUG -DB_ENDIAN -g -Wall -O::-D_REENTRANT:MACOSX::BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${ppc32_asm}:osx32:dlfcn:darwin-shared:-fPIC:-dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib", # iPhoneOS/iOS "iphoneos-cross","llvm-gcc:-O3 -isysroot \$(CROSS_TOP)/SDKs/\$(CROSS_SDK) -fomit-frame-pointer -fno-common::-D_REENTRANT:iOS:-Wl,-search_paths_first%:BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${no_asm}:dlfcn:darwin-shared:-fPIC -fno-common:-dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib", -@@ -1624,7 +1624,7 @@ +@@ -1629,7 +1629,7 @@ s/^CC=.*$/CC= $cc/; s/^AR=\s*ar/AR= $ar/; s/^RANLIB=.*/RANLIB= $ranlib/; @@ -34,9 +36,9 @@ } s/^CFLAG=.*$/CFLAG= $cflags/; s/^DEPFLAG=.*$/DEPFLAG=$depflags/; -diff util/domd ---- a/util/domd Fri Dec 05 01:24:16 2014 -0800 -+++ b/util/domd Fri Dec 05 01:52:29 2014 -0800 +diff -r 99ae439a07f1 util/domd +--- a/util/domd Fri Jan 09 12:50:43 2015 -0800 ++++ b/util/domd Fri Jan 09 12:53:52 2015 -0800 @@ -14,7 +14,7 @@ cp Makefile Makefile.save # fake the presence of Kerberos -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 22:34:20 2015 From: python-checkins at python.org (ned.deily) Date: Fri, 09 Jan 2015 21:34:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMjEy?= =?utf-8?q?=3A_2=2E7-specific_OS_X_installer_updates?= Message-ID: <20150109213420.8757.39544@psf.io> https://hg.python.org/cpython/rev/849ec86651b4 changeset: 94099:849ec86651b4 branch: 2.7 user: Ned Deily date: Fri Jan 09 13:29:04 2015 -0800 summary: Issue #23212: 2.7-specific OS X installer updates files: Mac/BuildScript/README.txt | 2 +- Mac/BuildScript/resources/ReadMe.rtf | 2 +- Misc/NEWS | 2 ++ 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Mac/BuildScript/README.txt b/Mac/BuildScript/README.txt --- a/Mac/BuildScript/README.txt +++ b/Mac/BuildScript/README.txt @@ -30,7 +30,7 @@ - builds the following third-party libraries - * libcrypto and libssl from OpenSSL 1.0.1j + * libcrypto and libssl from OpenSSL 1.0.1 * NCurses 5.9 * SQLite 3.7.13 * Oracle Sleepycat DB 4.8 (Python 2.x only) diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf --- a/Mac/BuildScript/resources/ReadMe.rtf +++ b/Mac/BuildScript/resources/ReadMe.rtf @@ -111,7 +111,7 @@ \i0 . To solve this problem, as of 2.7.9 the \i 10.5+ 32-bit-only python.org variant \i0 is linked with a private copy of -\i OpenSSL 1.0.1j +\i OpenSSL 1.0.1 \i0 ; it consults the same default certificate directory, \f1 /System/Library/OpenSSL \f0 . As before, it is still necessary to manage certificates yourself when you use this Python variant and, with certificate verification now enabled by default, you may now need to take additional steps to ensure your Python programs have access to CA certificates you trust. If you use this Python variant to build standalone applications with third-party tools like {\field{\*\fldinst{HYPERLINK "https://pypi.python.org/pypi/py2app/"}}{\fldrslt diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -92,6 +92,8 @@ - Issue #23032: Fix installer build failures on OS X 10.4 Tiger by disabling assembly code in the OpenSSL build. +- Issue #23212: Update 10.5 OS X installer build to use OpenSSL 1.0.1k. + What's New in Python 2.7.9? =========================== -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 22:34:21 2015 From: python-checkins at python.org (ned.deily) Date: Fri, 09 Jan 2015 21:34:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323212=3A_merge_from_3=2E4?= Message-ID: <20150109213421.11589.74242@psf.io> https://hg.python.org/cpython/rev/6d518aa5e1a2 changeset: 94102:6d518aa5e1a2 parent: 94097:27cbc877447b parent: 94101:726d67a7ebf2 user: Ned Deily date: Fri Jan 09 13:33:28 2015 -0800 summary: Issue #23212: merge from 3.4 files: Mac/BuildScript/build-installer.py | 6 +- Mac/BuildScript/openssl_sdk_makedepend.patch | 16 +++++---- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -237,9 +237,9 @@ result.extend([ dict( - name="OpenSSL 1.0.1j", - url="https://www.openssl.org/source/openssl-1.0.1j.tar.gz", - checksum='f7175c9cd3c39bb1907ac8bba9df8ed3', + name="OpenSSL 1.0.1k", + url="https://www.openssl.org/source/openssl-1.0.1k.tar.gz", + checksum='d4f002bd22a56881340105028842ae1f', patches=[ "openssl_sdk_makedepend.patch", ], diff --git a/Mac/BuildScript/openssl_sdk_makedepend.patch b/Mac/BuildScript/openssl_sdk_makedepend.patch --- a/Mac/BuildScript/openssl_sdk_makedepend.patch +++ b/Mac/BuildScript/openssl_sdk_makedepend.patch @@ -1,13 +1,15 @@ # openssl_sdk_makedepend.patch # -# using openssl 1.0.1j +# using openssl 1.0.1k # # - support building with an OS X SDK # - allow "make depend" to use compilers with names other than "gcc" diff Configure ---- a/Configure Fri Dec 05 01:24:16 2014 -0800 -+++ b/Configure Fri Dec 05 01:52:29 2014 -0800 + +diff -r 99ae439a07f1 Configure +--- a/Configure Fri Jan 09 12:50:43 2015 -0800 ++++ b/Configure Fri Jan 09 12:53:52 2015 -0800 @@ -577,11 +577,11 @@ ##### MacOS X (a.k.a. Rhapsody or Darwin) setup @@ -25,7 +27,7 @@ "debug-darwin-ppc-cc","cc:-DBN_DEBUG -DREF_CHECK -DCONF_DEBUG -DCRYPTO_MDEBUG -DB_ENDIAN -g -Wall -O::-D_REENTRANT:MACOSX::BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${ppc32_asm}:osx32:dlfcn:darwin-shared:-fPIC:-dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib", # iPhoneOS/iOS "iphoneos-cross","llvm-gcc:-O3 -isysroot \$(CROSS_TOP)/SDKs/\$(CROSS_SDK) -fomit-frame-pointer -fno-common::-D_REENTRANT:iOS:-Wl,-search_paths_first%:BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${no_asm}:dlfcn:darwin-shared:-fPIC -fno-common:-dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib", -@@ -1624,7 +1624,7 @@ +@@ -1629,7 +1629,7 @@ s/^CC=.*$/CC= $cc/; s/^AR=\s*ar/AR= $ar/; s/^RANLIB=.*/RANLIB= $ranlib/; @@ -34,9 +36,9 @@ } s/^CFLAG=.*$/CFLAG= $cflags/; s/^DEPFLAG=.*$/DEPFLAG=$depflags/; -diff util/domd ---- a/util/domd Fri Dec 05 01:24:16 2014 -0800 -+++ b/util/domd Fri Dec 05 01:52:29 2014 -0800 +diff -r 99ae439a07f1 util/domd +--- a/util/domd Fri Jan 09 12:50:43 2015 -0800 ++++ b/util/domd Fri Jan 09 12:53:52 2015 -0800 @@ -14,7 +14,7 @@ cp Makefile Makefile.save # fake the presence of Kerberos -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 22:34:20 2015 From: python-checkins at python.org (ned.deily) Date: Fri, 09 Jan 2015 21:34:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjEy?= =?utf-8?q?=3A_Update_OS_X_installer_build_OpenSSL_to_1=2E0=2E1k=2E?= Message-ID: <20150109213420.11561.76308@psf.io> https://hg.python.org/cpython/rev/ce3028357f8b changeset: 94100:ce3028357f8b branch: 3.4 parent: 94096:7438f2e30908 user: Ned Deily date: Fri Jan 09 13:29:54 2015 -0800 summary: Issue #23212: Update OS X installer build OpenSSL to 1.0.1k. (currently only used for builds with <= 10.5 deployment targets) files: Mac/BuildScript/build-installer.py | 6 +- Mac/BuildScript/openssl_sdk_makedepend.patch | 16 +++++---- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -237,9 +237,9 @@ result.extend([ dict( - name="OpenSSL 1.0.1j", - url="https://www.openssl.org/source/openssl-1.0.1j.tar.gz", - checksum='f7175c9cd3c39bb1907ac8bba9df8ed3', + name="OpenSSL 1.0.1k", + url="https://www.openssl.org/source/openssl-1.0.1k.tar.gz", + checksum='d4f002bd22a56881340105028842ae1f', patches=[ "openssl_sdk_makedepend.patch", ], diff --git a/Mac/BuildScript/openssl_sdk_makedepend.patch b/Mac/BuildScript/openssl_sdk_makedepend.patch --- a/Mac/BuildScript/openssl_sdk_makedepend.patch +++ b/Mac/BuildScript/openssl_sdk_makedepend.patch @@ -1,13 +1,15 @@ # openssl_sdk_makedepend.patch # -# using openssl 1.0.1j +# using openssl 1.0.1k # # - support building with an OS X SDK # - allow "make depend" to use compilers with names other than "gcc" diff Configure ---- a/Configure Fri Dec 05 01:24:16 2014 -0800 -+++ b/Configure Fri Dec 05 01:52:29 2014 -0800 + +diff -r 99ae439a07f1 Configure +--- a/Configure Fri Jan 09 12:50:43 2015 -0800 ++++ b/Configure Fri Jan 09 12:53:52 2015 -0800 @@ -577,11 +577,11 @@ ##### MacOS X (a.k.a. Rhapsody or Darwin) setup @@ -25,7 +27,7 @@ "debug-darwin-ppc-cc","cc:-DBN_DEBUG -DREF_CHECK -DCONF_DEBUG -DCRYPTO_MDEBUG -DB_ENDIAN -g -Wall -O::-D_REENTRANT:MACOSX::BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${ppc32_asm}:osx32:dlfcn:darwin-shared:-fPIC:-dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib", # iPhoneOS/iOS "iphoneos-cross","llvm-gcc:-O3 -isysroot \$(CROSS_TOP)/SDKs/\$(CROSS_SDK) -fomit-frame-pointer -fno-common::-D_REENTRANT:iOS:-Wl,-search_paths_first%:BN_LLONG RC4_CHAR RC4_CHUNK DES_UNROLL BF_PTR:${no_asm}:dlfcn:darwin-shared:-fPIC -fno-common:-dynamiclib:.\$(SHLIB_MAJOR).\$(SHLIB_MINOR).dylib", -@@ -1624,7 +1624,7 @@ +@@ -1629,7 +1629,7 @@ s/^CC=.*$/CC= $cc/; s/^AR=\s*ar/AR= $ar/; s/^RANLIB=.*/RANLIB= $ranlib/; @@ -34,9 +36,9 @@ } s/^CFLAG=.*$/CFLAG= $cflags/; s/^DEPFLAG=.*$/DEPFLAG=$depflags/; -diff util/domd ---- a/util/domd Fri Dec 05 01:24:16 2014 -0800 -+++ b/util/domd Fri Dec 05 01:52:29 2014 -0800 +diff -r 99ae439a07f1 util/domd +--- a/util/domd Fri Jan 09 12:50:43 2015 -0800 ++++ b/util/domd Fri Jan 09 12:53:52 2015 -0800 @@ -14,7 +14,7 @@ cp Makefile Makefile.save # fake the presence of Kerberos -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 22:34:21 2015 From: python-checkins at python.org (ned.deily) Date: Fri, 09 Jan 2015 21:34:21 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjEy?= =?utf-8?q?=3A_3=2E4-specific_OS_X_installer_updates?= Message-ID: <20150109213420.8761.38015@psf.io> https://hg.python.org/cpython/rev/726d67a7ebf2 changeset: 94101:726d67a7ebf2 branch: 3.4 user: Ned Deily date: Fri Jan 09 13:30:11 2015 -0800 summary: Issue #23212: 3.4-specific OS X installer updates files: Mac/BuildScript/README.txt | 46 +--------------- Mac/BuildScript/resources/ReadMe.rtf | 2 +- 2 files changed, 3 insertions(+), 45 deletions(-) diff --git a/Mac/BuildScript/README.txt b/Mac/BuildScript/README.txt --- a/Mac/BuildScript/README.txt +++ b/Mac/BuildScript/README.txt @@ -21,9 +21,6 @@ yet integrated into ``build-installer.py``. The steps prior to the flat package creation are the same as for 3.4.1 below. -For Python 3.4.0 and 3.4.1, PSF practice was to build two installer variants -for each release. - 1. 32-bit-only, i386 and PPC universal, capable on running on all machines supported by Mac OS X 10.5 through (at least) 10.9:: @@ -34,6 +31,7 @@ - builds the following third-party libraries + * libcrypto and libssl from OpenSSL 1.0.1 (new, as of 3.4.3) * NCurses 5.9 (http://bugs.python.org/issue15037) * SQLite 3.8.3.1 * XZ 5.0.5 @@ -75,6 +73,7 @@ - uses system-supplied versions of third-party libraries + * libcrypto and libssl from Apple OpenSSL 0.9.8 * readline module links with Apple BSD editline (libedit) - requires ActiveState Tcl/Tk 8.5.15.1 (or later) to be installed for building @@ -103,47 +102,6 @@ that the Xcode 3 gcc-4.2 compiler has had. -* For Python 2.7.x and 3.2.x, the 32-bit-only installer was configured to - support Mac OS X 10.3.9 through (at least) 10.6. Because it is - believed that there are few systems still running OS X 10.3 or 10.4 - and because it has become increasingly difficult to test and - support the differences in these earlier systems, as of Python 3.3.0 the PSF - 32-bit installer no longer supports them. For reference in building such - an installer yourself, the details are:: - - /usr/bin/python build-installer.py \ - --sdk-path=/Developer/SDKs/MacOSX10.4u.sdk \ - --universal-archs=32-bit \ - --dep-target=10.3 - - - builds the following third-party libraries - - * Bzip2 - * NCurses - * GNU Readline (GPL) - * SQLite 3 - * XZ - * Zlib 1.2.3 - * Oracle Sleepycat DB 4.8 (Python 2.x only) - - - requires ActiveState ``Tcl/Tk 8.4`` (currently 8.4.20) to be installed for building - - - recommended build environment: - - * Mac OS X 10.5.8 PPC or Intel - * Xcode 3.1.4 (or later) - * ``MacOSX10.4u`` SDK (later SDKs do not support PPC G3 processors) - * ``MACOSX_DEPLOYMENT_TARGET=10.3`` - * Apple ``gcc-4.0`` - * system Python 2.5 for documentation build with Sphinx - - - alternate build environments: - - * Mac OS X 10.6.8 with Xcode 3.2.6 - - need to change ``/System/Library/Frameworks/{Tcl,Tk}.framework/Version/Current`` to ``8.4`` - - - General Prerequisites --------------------- diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf --- a/Mac/BuildScript/resources/ReadMe.rtf +++ b/Mac/BuildScript/resources/ReadMe.rtf @@ -125,7 +125,7 @@ \i0 . To solve this problem, as of 3.4.3 the \i 10.5+ 32-bit-only python.org variant \i0 is linked with a private copy of -\i OpenSSL 1.0.1j +\i OpenSSL 1.0.1 \i0 ; it consults the same default certificate directory, \f1 /System/Library/OpenSSL \f0 . As before, it is still necessary to manage certificates yourself when you use this Python variant and, with certificate verification now enabled by default, you may now need to take additional steps to ensure your Python programs have access to CA certificates you trust. If you use this Python variant to build standalone applications with third-party tools like {\field{\*\fldinst{HYPERLINK "https://pypi.python.org/pypi/py2app/"}}{\fldrslt -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 23:41:10 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 09 Jan 2015 22:41:10 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_remove_buzzwor?= =?utf-8?q?d_=28closes_=2323210=29?= Message-ID: <20150109224044.11571.87094@psf.io> https://hg.python.org/cpython/rev/79f33147949b changeset: 94103:79f33147949b branch: 3.4 parent: 94101:726d67a7ebf2 user: Benjamin Peterson date: Fri Jan 09 16:40:23 2015 -0600 summary: remove buzzword (closes #23210) files: Objects/rangeobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -139,7 +139,7 @@ "range(stop) -> range object\n\ range(start, stop[, step]) -> range object\n\ \n\ -Return a virtual sequence of numbers from start to stop by step."); +Return a sequence of numbers from start to stop by step."); static void range_dealloc(rangeobject *r) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 9 23:41:10 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 09 Jan 2015 22:41:10 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMyMTAp?= Message-ID: <20150109224044.125890.56985@psf.io> https://hg.python.org/cpython/rev/154ae3af0317 changeset: 94104:154ae3af0317 parent: 94102:6d518aa5e1a2 parent: 94103:79f33147949b user: Benjamin Peterson date: Fri Jan 09 16:40:38 2015 -0600 summary: merge 3.4 (#23210) files: Objects/rangeobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -139,7 +139,7 @@ "range(stop) -> range object\n\ range(start, stop[, step]) -> range object\n\ \n\ -Return a virtual sequence of numbers from start to stop by step."); +Return a sequence of numbers from start to stop by step."); static void range_dealloc(rangeobject *r) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 10 00:21:38 2015 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 09 Jan 2015 23:21:38 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Add_PyCharm_reference=2E?= Message-ID: <20150109232129.8763.28265@psf.io> https://hg.python.org/peps/rev/0b4bc336a7fc changeset: 5669:0b4bc336a7fc user: Guido van Rossum date: Fri Jan 09 15:21:14 2015 -0800 summary: Add PyCharm reference. files: pep-0482.txt | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-) diff --git a/pep-0482.txt b/pep-0482.txt --- a/pep-0482.txt +++ b/pep-0482.txt @@ -38,6 +38,16 @@ Vitousek with Jeremy Siek and Jim Baker (the latter of Jython fame). +PyCharm +------- + +PyCharm by JetBrains has been providing a way to specify and check +types for about four years. The type system suggested by PyCharm +[pycharm]_ grew from simple class types to tuple types, generic types, +function types, etc. based on feedback of many users who shared their +experience of using type hints in their code. + + Others ------ @@ -196,6 +206,9 @@ .. [reticulated-paper] http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf +.. [pycharm] + https://github.com/JetBrains/python-skeletons#types + .. [obiwan] http://pypi.python.org/pypi/obiwan -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Sat Jan 10 09:00:46 2015 From: python-checkins at python.org (victor.stinner) Date: Sat, 10 Jan 2015 08:00:46 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319776=3A_Fix_test?= =?utf-8?q?=5Fpathlib=2Etest=5Fexpanduser=28=29?= Message-ID: <20150110080029.72577.69712@psf.io> https://hg.python.org/cpython/rev/63dac5212552 changeset: 94105:63dac5212552 user: Victor Stinner date: Sat Jan 10 09:00:20 2015 +0100 summary: Issue #19776: Fix test_pathlib.test_expanduser() Skip users with an empty home directory. files: Lib/test/test_pathlib.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1983,7 +1983,7 @@ for pwdent in pwd.getpwall(): othername = pwdent.pw_name otherhome = pwdent.pw_dir.rstrip('/') - if othername != username: + if othername != username and otherhome: break p1 = P('~/Documents') -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Sat Jan 10 09:35:38 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 10 Jan 2015 09:35:38 +0100 Subject: [Python-checkins] Daily reference leaks (154ae3af0317): sum=6 Message-ID: results for 154ae3af0317 on branch "default" -------------------------------------------- test_asyncio leaked [3, 0, 0] memory blocks, sum=3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogcGfLFP', '-x'] From solipsis at pitrou.net Sun Jan 11 09:47:52 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 11 Jan 2015 09:47:52 +0100 Subject: [Python-checkins] Daily reference leaks (63dac5212552): sum=1 Message-ID: results for 63dac5212552 on branch "default" -------------------------------------------- test_collections leaked [0, -2, 0] references, sum=-2 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogW9zElA', '-x'] From python-checkins at python.org Sun Jan 11 11:49:17 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 11 Jan 2015 10:49:17 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Removed_duplicated_dict_en?= =?utf-8?q?tries=2E?= Message-ID: <20150111104856.125902.48174@psf.io> https://hg.python.org/cpython/rev/5b2212e0c350 changeset: 94106:5b2212e0c350 user: Serhiy Storchaka date: Sun Jan 11 12:48:17 2015 +0200 summary: Removed duplicated dict entries. files: Lib/tokenize.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/tokenize.py b/Lib/tokenize.py --- a/Lib/tokenize.py +++ b/Lib/tokenize.py @@ -186,7 +186,6 @@ "rB'''": Single3, 'rB"""': Double3, "RB'''": Single3, 'RB"""': Double3, "u'''": Single3, 'u"""': Double3, - "R'''": Single3, 'R"""': Double3, "U'''": Single3, 'U"""': Double3, 'r': None, 'R': None, 'b': None, 'B': None, 'u': None, 'U': None} -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 12:55:38 2015 From: python-checkins at python.org (mark.dickinson) Date: Sun, 11 Jan 2015 11:55:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323185=3A_add_math?= =?utf-8?q?=2Einf_and_math=2Enan_constants=2E?= Message-ID: <20150111115537.125892.41308@psf.io> https://hg.python.org/cpython/rev/cf4bf577749c changeset: 94107:cf4bf577749c user: Mark Dickinson date: Sun Jan 11 11:55:29 2015 +0000 summary: Issue #23185: add math.inf and math.nan constants. files: Doc/library/math.rst | 16 ++++++++++++++ Doc/whatsnew/3.5.rst | 6 +++++ Lib/test/test_math.py | 11 +++++++++ Misc/NEWS | 2 + Modules/mathmodule.c | 35 ++++++++++++++++++++++++++++++- 5 files changed, 69 insertions(+), 1 deletions(-) diff --git a/Doc/library/math.rst b/Doc/library/math.rst --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -383,6 +383,22 @@ The mathematical constant e = 2.718281..., to available precision. +.. data:: inf + + A floating-point positive infinity. (For negative infinity, use + ``-math.inf``.) Equivalent to the output of ``float('inf')``. + + .. versionadded:: 3.5 + + +.. data:: nan + + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. + + .. versionadded:: 3.5 + + .. impl-detail:: The :mod:`math` module consists mostly of thin wrappers around the platform C diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -243,6 +243,12 @@ * Now unmatched groups are replaced with empty strings in :func:`re.sub` and :func:`re.subn`. (Contributed by Serhiy Storchaka in :issue:`1519638`.) +math +---- + +* :data:`math.inf` and :data:`math.nan` constants added. (Contributed by Mark + Dickinson in :issue:`23185`.) + shutil ------ diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py --- a/Lib/test/test_math.py +++ b/Lib/test/test_math.py @@ -983,6 +983,17 @@ self.assertFalse(math.isinf(0.)) self.assertFalse(math.isinf(1.)) + @requires_IEEE_754 + def test_nan_constant(self): + self.assertTrue(math.isnan(math.nan)) + + @requires_IEEE_754 + def test_inf_constant(self): + self.assertTrue(math.isinf(math.inf)) + self.assertGreater(math.inf, 0.0) + self.assertEqual(math.inf, float("inf")) + self.assertEqual(-math.inf, float("-inf")) + # RED_FLAG 16-Oct-2000 Tim # While 2.0 is more consistent about exceptions than previous releases, it # still fails this part of the test on some platforms. For now, we only diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,8 @@ Library ------- +- Issue #23185: Add math.inf and math.nan constants. + - Issue #23186: Add ssl.SSLObject.shared_ciphers() and ssl.SSLSocket.shared_ciphers() to fetch the client's list ciphers sent at handshake. diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -223,6 +223,35 @@ return num/den; } +/* Constant for +infinity, generated in the same way as float('inf'). */ + +static double +m_inf(void) +{ +#ifndef PY_NO_SHORT_FLOAT_REPR + return _Py_dg_infinity(0); +#else + return Py_HUGE_VAL; +#endif +} + +/* Constant nan value, generated in the same way as float('nan'). */ +/* We don't currently assume that Py_NAN is defined everywhere. */ + +#if !defined(PY_NO_SHORT_FLOAT_REPR) || defined(Py_NAN) + +static double +m_nan(void) +{ +#ifndef PY_NO_SHORT_FLOAT_REPR + return _Py_dg_stdnan(0); +#else + return Py_NAN; +#endif +} + +#endif + static double m_tgamma(double x) { @@ -2009,7 +2038,11 @@ PyModule_AddObject(m, "pi", PyFloat_FromDouble(Py_MATH_PI)); PyModule_AddObject(m, "e", PyFloat_FromDouble(Py_MATH_E)); + PyModule_AddObject(m, "inf", PyFloat_FromDouble(m_inf())); +#if !defined(PY_NO_SHORT_FLOAT_REPR) || defined(Py_NAN) + PyModule_AddObject(m, "nan", PyFloat_FromDouble(m_nan())); +#endif - finally: + finally: return m; } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 14:03:52 2015 From: python-checkins at python.org (mark.dickinson) Date: Sun, 11 Jan 2015 13:03:52 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxOTAy?= =?utf-8?q?=3A_Replace_incorrect_=27hyperbolic_arc_sine=27_=28etc=2E=29_wi?= =?utf-8?q?th_=27inverse?= Message-ID: <20150111130344.125882.28696@psf.io> https://hg.python.org/cpython/rev/d81cabb39de3 changeset: 94108:d81cabb39de3 branch: 2.7 parent: 94099:849ec86651b4 user: Mark Dickinson date: Sun Jan 11 13:03:06 2015 +0000 summary: Issue #21902: Replace incorrect 'hyperbolic arc sine' (etc.) with 'inverse hyperbolic sine' (etc.). Remove meaningless reference to radians. files: Doc/library/cmath.rst | 8 ++++---- Modules/cmathmodule.c | 6 +++--- Modules/mathmodule.c | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -161,13 +161,13 @@ .. function:: acosh(x) - Return the hyperbolic arc cosine of *x*. There is one branch cut, extending left - from 1 along the real axis to -?, continuous from above. + Return the inverse hyperbolic cosine of *x*. There is one branch cut, + extending left from 1 along the real axis to -?, continuous from above. .. function:: asinh(x) - Return the hyperbolic arc sine of *x*. There are two branch cuts: + Return the inverse hyperbolic sine of *x*. There are two branch cuts: One extends from ``1j`` along the imaginary axis to ``?j``, continuous from the right. The other extends from ``-1j`` along the imaginary axis to ``-?j``, continuous from the left. @@ -178,7 +178,7 @@ .. function:: atanh(x) - Return the hyperbolic arc tangent of *x*. There are two branch cuts: One + Return the inverse hyperbolic tangent of *x*. There are two branch cuts: One extends from ``1`` along the real axis to ``?``, continuous from below. The other extends from ``-1`` along the real axis to ``-?``, continuous from above. diff --git a/Modules/cmathmodule.c b/Modules/cmathmodule.c --- a/Modules/cmathmodule.c +++ b/Modules/cmathmodule.c @@ -192,7 +192,7 @@ PyDoc_STRVAR(c_acosh_doc, "acosh(x)\n" "\n" -"Return the hyperbolic arccosine of x."); +"Return the inverse hyperbolic cosine of x."); static Py_complex @@ -249,7 +249,7 @@ PyDoc_STRVAR(c_asinh_doc, "asinh(x)\n" "\n" -"Return the hyperbolic arc sine of x."); +"Return the inverse hyperbolic sine of x."); static Py_complex @@ -353,7 +353,7 @@ PyDoc_STRVAR(c_atanh_doc, "atanh(x)\n" "\n" -"Return the hyperbolic arc tangent of x."); +"Return the inverse hyperbolic tangent of x."); static Py_complex diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -809,18 +809,18 @@ FUNC1(acos, acos, 0, "acos(x)\n\nReturn the arc cosine (measured in radians) of x.") FUNC1(acosh, m_acosh, 0, - "acosh(x)\n\nReturn the hyperbolic arc cosine (measured in radians) of x.") + "acosh(x)\n\nReturn the inverse hyperbolic cosine of x.") FUNC1(asin, asin, 0, "asin(x)\n\nReturn the arc sine (measured in radians) of x.") FUNC1(asinh, m_asinh, 0, - "asinh(x)\n\nReturn the hyperbolic arc sine (measured in radians) of x.") + "asinh(x)\n\nReturn the inverse hyperbolic sine of x.") FUNC1(atan, atan, 0, "atan(x)\n\nReturn the arc tangent (measured in radians) of x.") FUNC2(atan2, m_atan2, "atan2(y, x)\n\nReturn the arc tangent (measured in radians) of y/x.\n" "Unlike atan(y/x), the signs of both x and y are considered.") FUNC1(atanh, m_atanh, 0, - "atanh(x)\n\nReturn the hyperbolic arc tangent (measured in radians) of x.") + "atanh(x)\n\nReturn the inverse hyperbolic tangent of x.") FUNC1(ceil, ceil, 0, "ceil(x)\n\nReturn the ceiling of x as a float.\n" "This is the smallest integral value >= x.") -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 14:23:44 2015 From: python-checkins at python.org (mark.dickinson) Date: Sun, 11 Jan 2015 13:23:44 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxOTAy?= =?utf-8?q?=3A_Replace_incorrect_=27hyperbolic_arc_sine=27_=28etc=2E=29_wi?= =?utf-8?q?th_=27inverse?= Message-ID: <20150111132335.8745.27549@psf.io> https://hg.python.org/cpython/rev/5edfc6c929f9 changeset: 94109:5edfc6c929f9 branch: 3.4 parent: 94103:79f33147949b user: Mark Dickinson date: Sun Jan 11 13:08:05 2015 +0000 summary: Issue #21902: Replace incorrect 'hyperbolic arc sine' (etc.) with 'inverse hyperbolic sine' (etc.). Remove meaningless reference to radians. files: Doc/library/cmath.rst | 8 ++++---- Modules/cmathmodule.c | 6 +++--- Modules/mathmodule.c | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -149,13 +149,13 @@ .. function:: acosh(x) - Return the hyperbolic arc cosine of *x*. There is one branch cut, extending left - from 1 along the real axis to -?, continuous from above. + Return the inverse hyperbolic cosine of *x*. There is one branch cut, + extending left from 1 along the real axis to -?, continuous from above. .. function:: asinh(x) - Return the hyperbolic arc sine of *x*. There are two branch cuts: + Return the inverse hyperbolic sine of *x*. There are two branch cuts: One extends from ``1j`` along the imaginary axis to ``?j``, continuous from the right. The other extends from ``-1j`` along the imaginary axis to ``-?j``, continuous from the left. @@ -163,7 +163,7 @@ .. function:: atanh(x) - Return the hyperbolic arc tangent of *x*. There are two branch cuts: One + Return the inverse hyperbolic tangent of *x*. There are two branch cuts: One extends from ``1`` along the real axis to ``?``, continuous from below. The other extends from ``-1`` along the real axis to ``-?``, continuous from above. diff --git a/Modules/cmathmodule.c b/Modules/cmathmodule.c --- a/Modules/cmathmodule.c +++ b/Modules/cmathmodule.c @@ -192,7 +192,7 @@ PyDoc_STRVAR(c_acosh_doc, "acosh(x)\n" "\n" -"Return the hyperbolic arccosine of x."); +"Return the inverse hyperbolic cosine of x."); static Py_complex @@ -249,7 +249,7 @@ PyDoc_STRVAR(c_asinh_doc, "asinh(x)\n" "\n" -"Return the hyperbolic arc sine of x."); +"Return the inverse hyperbolic sine of x."); static Py_complex @@ -353,7 +353,7 @@ PyDoc_STRVAR(c_atanh_doc, "atanh(x)\n" "\n" -"Return the hyperbolic arc tangent of x."); +"Return the inverse hyperbolic tangent of x."); static Py_complex diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -873,18 +873,18 @@ FUNC1(acos, acos, 0, "acos(x)\n\nReturn the arc cosine (measured in radians) of x.") FUNC1(acosh, m_acosh, 0, - "acosh(x)\n\nReturn the hyperbolic arc cosine (measured in radians) of x.") + "acosh(x)\n\nReturn the inverse hyperbolic cosine of x.") FUNC1(asin, asin, 0, "asin(x)\n\nReturn the arc sine (measured in radians) of x.") FUNC1(asinh, m_asinh, 0, - "asinh(x)\n\nReturn the hyperbolic arc sine (measured in radians) of x.") + "asinh(x)\n\nReturn the inverse hyperbolic sine of x.") FUNC1(atan, atan, 0, "atan(x)\n\nReturn the arc tangent (measured in radians) of x.") FUNC2(atan2, m_atan2, "atan2(y, x)\n\nReturn the arc tangent (measured in radians) of y/x.\n" "Unlike atan(y/x), the signs of both x and y are considered.") FUNC1(atanh, m_atanh, 0, - "atanh(x)\n\nReturn the hyperbolic arc tangent (measured in radians) of x.") + "atanh(x)\n\nReturn the inverse hyperbolic tangent of x.") static PyObject * math_ceil(PyObject *self, PyObject *number) { _Py_IDENTIFIER(__ceil__); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 14:23:44 2015 From: python-checkins at python.org (mark.dickinson) Date: Sun, 11 Jan 2015 13:23:44 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2321092=3A_Merge_from_3=2E4=2E?= Message-ID: <20150111132335.125882.8030@psf.io> https://hg.python.org/cpython/rev/36099a05d76a changeset: 94110:36099a05d76a parent: 94107:cf4bf577749c parent: 94109:5edfc6c929f9 user: Mark Dickinson date: Sun Jan 11 13:22:44 2015 +0000 summary: Issue #21092: Merge from 3.4. files: Doc/library/cmath.rst | 8 ++++---- Modules/clinic/cmathmodule.c.h | 6 +++--- Modules/cmathmodule.c | 6 +++--- Modules/mathmodule.c | 6 +++--- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -149,13 +149,13 @@ .. function:: acosh(x) - Return the hyperbolic arc cosine of *x*. There is one branch cut, extending left - from 1 along the real axis to -?, continuous from above. + Return the inverse hyperbolic cosine of *x*. There is one branch cut, + extending left from 1 along the real axis to -?, continuous from above. .. function:: asinh(x) - Return the hyperbolic arc sine of *x*. There are two branch cuts: + Return the inverse hyperbolic sine of *x*. There are two branch cuts: One extends from ``1j`` along the imaginary axis to ``?j``, continuous from the right. The other extends from ``-1j`` along the imaginary axis to ``-?j``, continuous from the left. @@ -163,7 +163,7 @@ .. function:: atanh(x) - Return the hyperbolic arc tangent of *x*. There are two branch cuts: One + Return the inverse hyperbolic tangent of *x*. There are two branch cuts: One extends from ``1`` along the real axis to ``?``, continuous from below. The other extends from ``-1`` along the real axis to ``-?``, continuous from above. diff --git a/Modules/clinic/cmathmodule.c.h b/Modules/clinic/cmathmodule.c.h --- a/Modules/clinic/cmathmodule.c.h +++ b/Modules/clinic/cmathmodule.c.h @@ -49,7 +49,7 @@ "acosh($module, z, /)\n" "--\n" "\n" -"Return the hyperbolic arccosine of z."); +"Return the inverse hyperbolic cosine of z."); #define CMATH_ACOSH_METHODDEF \ {"acosh", (PyCFunction)cmath_acosh, METH_VARARGS, cmath_acosh__doc__}, @@ -135,7 +135,7 @@ "asinh($module, z, /)\n" "--\n" "\n" -"Return the hyperbolic arc sine of z."); +"Return the inverse hyperbolic sine of z."); #define CMATH_ASINH_METHODDEF \ {"asinh", (PyCFunction)cmath_asinh, METH_VARARGS, cmath_asinh__doc__}, @@ -221,7 +221,7 @@ "atanh($module, z, /)\n" "--\n" "\n" -"Return the hyperbolic arc tangent of z."); +"Return the inverse hyperbolic tangent of z."); #define CMATH_ATANH_METHODDEF \ {"atanh", (PyCFunction)cmath_atanh, METH_VARARGS, cmath_atanh__doc__}, diff --git a/Modules/cmathmodule.c b/Modules/cmathmodule.c --- a/Modules/cmathmodule.c +++ b/Modules/cmathmodule.c @@ -207,7 +207,7 @@ /*[clinic input] cmath.acosh = cmath.acos -Return the hyperbolic arccosine of z. +Return the inverse hyperbolic cosine of z. [clinic start generated code]*/ static Py_complex @@ -262,7 +262,7 @@ /*[clinic input] cmath.asinh = cmath.acos -Return the hyperbolic arc sine of z. +Return the inverse hyperbolic sine of z. [clinic start generated code]*/ static Py_complex @@ -353,7 +353,7 @@ /*[clinic input] cmath.atanh = cmath.acos -Return the hyperbolic arc tangent of z. +Return the inverse hyperbolic tangent of z. [clinic start generated code]*/ static Py_complex diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -902,18 +902,18 @@ FUNC1(acos, acos, 0, "acos(x)\n\nReturn the arc cosine (measured in radians) of x.") FUNC1(acosh, m_acosh, 0, - "acosh(x)\n\nReturn the hyperbolic arc cosine (measured in radians) of x.") + "acosh(x)\n\nReturn the inverse hyperbolic cosine of x.") FUNC1(asin, asin, 0, "asin(x)\n\nReturn the arc sine (measured in radians) of x.") FUNC1(asinh, m_asinh, 0, - "asinh(x)\n\nReturn the hyperbolic arc sine (measured in radians) of x.") + "asinh(x)\n\nReturn the inverse hyperbolic sine of x.") FUNC1(atan, atan, 0, "atan(x)\n\nReturn the arc tangent (measured in radians) of x.") FUNC2(atan2, m_atan2, "atan2(y, x)\n\nReturn the arc tangent (measured in radians) of y/x.\n" "Unlike atan(y/x), the signs of both x and y are considered.") FUNC1(atanh, m_atanh, 0, - "atanh(x)\n\nReturn the hyperbolic arc tangent (measured in radians) of x.") + "atanh(x)\n\nReturn the inverse hyperbolic tangent of x.") static PyObject * math_ceil(PyObject *self, PyObject *number) { _Py_IDENTIFIER(__ceil__); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 15:07:08 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 11 Jan 2015 14:07:08 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322952=3A_improve_multiprocessing_doc_introducti?= =?utf-8?q?on_and_defer_notes_until?= Message-ID: <20150111140707.11591.10135@psf.io> https://hg.python.org/cpython/rev/a65c23ea5f9e changeset: 94112:a65c23ea5f9e parent: 94110:36099a05d76a parent: 94111:a9a9c71f8e15 user: Antoine Pitrou date: Sun Jan 11 15:06:39 2015 +0100 summary: Issue #22952: improve multiprocessing doc introduction and defer notes until appropriate. Patch by Davin Potts. files: Doc/library/multiprocessing.rst | 92 +++++++++++++------- 1 files changed, 57 insertions(+), 35 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -16,41 +16,27 @@ leverage multiple processors on a given machine. It runs on both Unix and Windows. -.. note:: - - Some of this package's functionality requires a functioning shared semaphore - implementation on the host operating system. Without one, the - :mod:`multiprocessing.synchronize` module will be disabled, and attempts to - import it will result in an :exc:`ImportError`. See - :issue:`3770` for additional information. - -.. note:: - - Functionality within this package requires that the ``__main__`` module be - importable by the children. This is covered in :ref:`multiprocessing-programming` - however it is worth pointing out here. This means that some examples, such - as the :class:`multiprocessing.pool.Pool` examples will not work in the - interactive interpreter. For example:: - - >>> from multiprocessing import Pool - >>> p = Pool(5) - >>> def f(x): - ... return x*x - ... - >>> p.map(f, [1,2,3]) - Process PoolWorker-1: - Process PoolWorker-2: - Process PoolWorker-3: - Traceback (most recent call last): - Traceback (most recent call last): - Traceback (most recent call last): - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - - (If you try this it will actually output three full tracebacks - interleaved in a semi-random fashion, and then you may have to - stop the master process somehow.) +The :mod:`multiprocessing` module also introduces APIs which do not have +analogs in the :mod:`threading` module. A prime example of this is the +:class:`~multiprocessing.pool.Pool` object which offers a convenient means of +parallelizing the execution of a function across multiple input values, +distributing the input data across processes (data parallelism). The following +example demonstrates the common practice of defining such functions in a module +so that child processes can successfully import that module. This basic example +of data parallelism using :class:`~multiprocessing.pool.Pool`, :: + + from multiprocessing import Pool + + def f(x): + return x*x + + if __name__ == '__main__': + with Pool(5) as p: + print(p.map(f, [1, 2, 3])) + +will print to standard output :: + + [1, 4, 9] The :class:`Process` class @@ -276,6 +262,14 @@ Without using the lock output from the different processes is liable to get all mixed up. +.. note:: + + Some of this package's functionality requires a functioning shared semaphore + implementation on the host operating system. Without one, the + :mod:`multiprocessing.synchronize` module will be disabled, and attempts to + import it will result in an :exc:`ImportError`. See + :issue:`3770` for additional information. + Sharing state between processes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -406,6 +400,34 @@ Note that the methods of a pool should only ever be used by the process which created it. +.. note:: + + Functionality within this package requires that the ``__main__`` module be + importable by the children. This is covered in :ref:`multiprocessing-programming` + however it is worth pointing out here. This means that some examples, such + as the :class:`multiprocessing.pool.Pool` examples will not work in the + interactive interpreter. For example:: + + >>> from multiprocessing import Pool + >>> p = Pool(5) + >>> def f(x): + ... return x*x + ... + >>> p.map(f, [1,2,3]) + Process PoolWorker-1: + Process PoolWorker-2: + Process PoolWorker-3: + Traceback (most recent call last): + Traceback (most recent call last): + Traceback (most recent call last): + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + + (If you try this it will actually output three full tracebacks + interleaved in a semi-random fashion, and then you may have to + stop the master process somehow.) + Reference --------- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 15:07:08 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 11 Jan 2015 14:07:08 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyOTUy?= =?utf-8?q?=3A_improve_multiprocessing_doc_introduction_and_defer_notes_un?= =?utf-8?q?til?= Message-ID: <20150111140707.125904.97891@psf.io> https://hg.python.org/cpython/rev/a9a9c71f8e15 changeset: 94111:a9a9c71f8e15 branch: 3.4 parent: 94109:5edfc6c929f9 user: Antoine Pitrou date: Sun Jan 11 15:05:29 2015 +0100 summary: Issue #22952: improve multiprocessing doc introduction and defer notes until appropriate. Patch by Davin Potts. files: Doc/library/multiprocessing.rst | 92 +++++++++++++------- 1 files changed, 57 insertions(+), 35 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -16,41 +16,27 @@ leverage multiple processors on a given machine. It runs on both Unix and Windows. -.. note:: - - Some of this package's functionality requires a functioning shared semaphore - implementation on the host operating system. Without one, the - :mod:`multiprocessing.synchronize` module will be disabled, and attempts to - import it will result in an :exc:`ImportError`. See - :issue:`3770` for additional information. - -.. note:: - - Functionality within this package requires that the ``__main__`` module be - importable by the children. This is covered in :ref:`multiprocessing-programming` - however it is worth pointing out here. This means that some examples, such - as the :class:`multiprocessing.pool.Pool` examples will not work in the - interactive interpreter. For example:: - - >>> from multiprocessing import Pool - >>> p = Pool(5) - >>> def f(x): - ... return x*x - ... - >>> p.map(f, [1,2,3]) - Process PoolWorker-1: - Process PoolWorker-2: - Process PoolWorker-3: - Traceback (most recent call last): - Traceback (most recent call last): - Traceback (most recent call last): - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - - (If you try this it will actually output three full tracebacks - interleaved in a semi-random fashion, and then you may have to - stop the master process somehow.) +The :mod:`multiprocessing` module also introduces APIs which do not have +analogs in the :mod:`threading` module. A prime example of this is the +:class:`~multiprocessing.pool.Pool` object which offers a convenient means of +parallelizing the execution of a function across multiple input values, +distributing the input data across processes (data parallelism). The following +example demonstrates the common practice of defining such functions in a module +so that child processes can successfully import that module. This basic example +of data parallelism using :class:`~multiprocessing.pool.Pool`, :: + + from multiprocessing import Pool + + def f(x): + return x*x + + if __name__ == '__main__': + with Pool(5) as p: + print(p.map(f, [1, 2, 3])) + +will print to standard output :: + + [1, 4, 9] The :class:`Process` class @@ -276,6 +262,14 @@ Without using the lock output from the different processes is liable to get all mixed up. +.. note:: + + Some of this package's functionality requires a functioning shared semaphore + implementation on the host operating system. Without one, the + :mod:`multiprocessing.synchronize` module will be disabled, and attempts to + import it will result in an :exc:`ImportError`. See + :issue:`3770` for additional information. + Sharing state between processes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -406,6 +400,34 @@ Note that the methods of a pool should only ever be used by the process which created it. +.. note:: + + Functionality within this package requires that the ``__main__`` module be + importable by the children. This is covered in :ref:`multiprocessing-programming` + however it is worth pointing out here. This means that some examples, such + as the :class:`multiprocessing.pool.Pool` examples will not work in the + interactive interpreter. For example:: + + >>> from multiprocessing import Pool + >>> p = Pool(5) + >>> def f(x): + ... return x*x + ... + >>> p.map(f, [1,2,3]) + Process PoolWorker-1: + Process PoolWorker-2: + Process PoolWorker-3: + Traceback (most recent call last): + Traceback (most recent call last): + Traceback (most recent call last): + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + + (If you try this it will actually output three full tracebacks + interleaved in a semi-random fashion, and then you may have to + stop the master process somehow.) + Reference --------- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 15:09:38 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 11 Jan 2015 14:09:38 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIyOTUy?= =?utf-8?q?=3A_improve_multiprocessing_doc_introduction_and_defer_notes_un?= =?utf-8?q?til?= Message-ID: <20150111140935.125896.30028@psf.io> https://hg.python.org/cpython/rev/e7d03a33e675 changeset: 94113:e7d03a33e675 branch: 2.7 parent: 94108:d81cabb39de3 user: Antoine Pitrou date: Sun Jan 11 15:09:27 2015 +0100 summary: Issue #22952: improve multiprocessing doc introduction and defer notes until appropriate. Patch by Davin Potts. files: Doc/library/multiprocessing.rst | 93 ++++++++++++-------- 1 files changed, 57 insertions(+), 36 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -18,41 +18,27 @@ leverage multiple processors on a given machine. It runs on both Unix and Windows. -.. warning:: - - Some of this package's functionality requires a functioning shared semaphore - implementation on the host operating system. Without one, the - :mod:`multiprocessing.synchronize` module will be disabled, and attempts to - import it will result in an :exc:`ImportError`. See - :issue:`3770` for additional information. - -.. note:: - - Functionality within this package requires that the ``__main__`` module be - importable by the children. This is covered in :ref:`multiprocessing-programming` - however it is worth pointing out here. This means that some examples, such - as the :class:`multiprocessing.Pool` examples will not work in the - interactive interpreter. For example:: - - >>> from multiprocessing import Pool - >>> p = Pool(5) - >>> def f(x): - ... return x*x - ... - >>> p.map(f, [1,2,3]) - Process PoolWorker-1: - Process PoolWorker-2: - Process PoolWorker-3: - Traceback (most recent call last): - Traceback (most recent call last): - Traceback (most recent call last): - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - - (If you try this it will actually output three full tracebacks - interleaved in a semi-random fashion, and then you may have to - stop the master process somehow.) +The :mod:`multiprocessing` module also introduces APIs which do not have +analogs in the :mod:`threading` module. A prime example of this is the +:class:`Pool` object which offers a convenient means of parallelizing the +execution of a function across multiple input values, distributing the +input data across processes (data parallelism). The following example +demonstrates the common practice of defining such functions in a module so +that child processes can successfully import that module. This basic example +of data parallelism using :class:`Pool`, :: + + from multiprocessing import Pool + + def f(x): + return x*x + + if __name__ == '__main__': + p = Pool(5) + print(p.map(f, [1, 2, 3])) + +will print to standard output :: + + [1, 4, 9] The :class:`Process` class @@ -99,7 +85,6 @@ necessary, see :ref:`multiprocessing-programming`. - Exchanging objects between processes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -175,6 +160,14 @@ Without using the lock output from the different processes is liable to get all mixed up. +.. warning:: + + Some of this package's functionality requires a functioning shared semaphore + implementation on the host operating system. Without one, the + :mod:`multiprocessing.synchronize` module will be disabled, and attempts to + import it will result in an :exc:`ImportError`. See + :issue:`3770` for additional information. + Sharing state between processes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -290,6 +283,34 @@ Note that the methods of a pool should only ever be used by the process which created it. +.. note:: + + Functionality within this package requires that the ``__main__`` module be + importable by the children. This is covered in :ref:`multiprocessing-programming` + however it is worth pointing out here. This means that some examples, such + as the :class:`Pool` examples will not work in the interactive interpreter. + For example:: + + >>> from multiprocessing import Pool + >>> p = Pool(5) + >>> def f(x): + ... return x*x + ... + >>> p.map(f, [1,2,3]) + Process PoolWorker-1: + Process PoolWorker-2: + Process PoolWorker-3: + Traceback (most recent call last): + Traceback (most recent call last): + Traceback (most recent call last): + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + + (If you try this it will actually output three full tracebacks + interleaved in a semi-random fashion, and then you may have to + stop the master process somehow.) + Reference --------- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 21:22:26 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 11 Jan 2015 20:22:26 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_remove_extra_definite_arti?= =?utf-8?q?cle?= Message-ID: <20150111202226.11591.86096@psf.io> https://hg.python.org/cpython/rev/ca5fe2148490 changeset: 94115:ca5fe2148490 user: Benjamin Peterson date: Sun Jan 11 15:22:07 2015 -0500 summary: remove extra definite article files: Lib/ssl.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -573,8 +573,8 @@ return self._sslobj.cipher() def shared_ciphers(self): - """Return the a list of ciphers shared by the client during the - handshake or None if this is not a valid server connection. + """Return a list of ciphers shared by the client during the handshake or + None if this is not a valid server connection. """ return self._sslobj.shared_ciphers() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 21:49:48 2015 From: python-checkins at python.org (donald.stufft) Date: Sun, 11 Jan 2015 20:49:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Bump_setuptool?= =?utf-8?q?s_to_11=2E3=2E1?= Message-ID: <20150111204943.8753.8140@psf.io> https://hg.python.org/cpython/rev/ad92f9b2fe04 changeset: 94116:ad92f9b2fe04 branch: 2.7 parent: 94113:e7d03a33e675 user: Donald Stufft date: Sun Jan 11 15:49:22 2015 -0500 summary: Bump setuptools to 11.3.1 files: Lib/ensurepip/__init__.py | 2 +- Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl | Bin 3 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -12,7 +12,7 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "11.0" +_SETUPTOOLS_VERSION = "11.3.1" _PIP_VERSION = "6.0.6" diff --git a/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl deleted file mode 100644 index ceeaa3cd5c326495156cb90ed3934ee5b712ad2e..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..6892ff89b4f916c40a0044c18d4b41dbf73340e3 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 21:51:19 2015 From: python-checkins at python.org (donald.stufft) Date: Sun, 11 Jan 2015 20:51:19 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Update_setupto?= =?utf-8?q?ols_to_11=2E3=2E1?= Message-ID: <20150111205118.72575.47284@psf.io> https://hg.python.org/cpython/rev/1b145e8ae4be changeset: 94117:1b145e8ae4be branch: 3.4 parent: 94111:a9a9c71f8e15 user: Donald Stufft date: Sun Jan 11 15:51:11 2015 -0500 summary: Update setuptools to 11.3.1 files: Lib/ensurepip/__init__.py | 2 +- Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl | Bin 3 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -8,7 +8,7 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "11.0" +_SETUPTOOLS_VERSION = "11.3.1" _PIP_VERSION = "6.0.6" diff --git a/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl deleted file mode 100644 index ceeaa3cd5c326495156cb90ed3934ee5b712ad2e..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..6892ff89b4f916c40a0044c18d4b41dbf73340e3 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 11 21:53:26 2015 From: python-checkins at python.org (donald.stufft) Date: Sun, 11 Jan 2015 20:53:26 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merged_3=2E4_into_default?= Message-ID: <20150111205326.72575.34916@psf.io> https://hg.python.org/cpython/rev/8b3c609f3f73 changeset: 94118:8b3c609f3f73 parent: 94115:ca5fe2148490 parent: 94117:1b145e8ae4be user: Donald Stufft date: Sun Jan 11 15:53:02 2015 -0500 summary: Merged 3.4 into default files: Lib/ensurepip/__init__.py | 2 +- Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl | Bin 3 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -8,7 +8,7 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "11.0" +_SETUPTOOLS_VERSION = "11.3.1" _PIP_VERSION = "6.0.6" diff --git a/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.0-py2.py3-none-any.whl deleted file mode 100644 index ceeaa3cd5c326495156cb90ed3934ee5b712ad2e..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..6892ff89b4f916c40a0044c18d4b41dbf73340e3 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Mon Jan 12 08:53:57 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 12 Jan 2015 08:53:57 +0100 Subject: [Python-checkins] Daily reference leaks (8b3c609f3f73): sum=-3 Message-ID: results for 8b3c609f3f73 on branch "default" -------------------------------------------- test_collections leaked [-4, 0, 0] references, sum=-4 test_collections leaked [-2, 0, 0] memory blocks, sum=-2 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKzw4VR', '-x'] From python-checkins at python.org Mon Jan 12 10:58:57 2015 From: python-checkins at python.org (larry.hastings) Date: Mon, 12 Jan 2015 09:58:57 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Removed_Ronald_Oussoren_from_?= =?utf-8?q?being_listed_as_Mac_PE_for_3=2E4_and_3=2E5=2E?= Message-ID: <20150112095856.72571.9080@psf.io> https://hg.python.org/peps/rev/59dfaaf19a14 changeset: 5670:59dfaaf19a14 user: Larry Hastings date: Mon Jan 12 01:58:52 2015 -0800 summary: Removed Ronald Oussoren from being listed as Mac PE for 3.4 and 3.5. Ned has been the only Mac PE for 3.4, and will be again for 3.5. files: pep-0429.txt | 2 +- pep-0478.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0429.txt b/pep-0429.txt --- a/pep-0429.txt +++ b/pep-0429.txt @@ -27,7 +27,7 @@ - 3.4 Release Manager: Larry Hastings - Windows installers: Martin v. L?wis -- Mac installers: Ned Deily / Ronald Oussoren +- Mac installers: Ned Deily - Documentation: Georg Brandl diff --git a/pep-0478.txt b/pep-0478.txt --- a/pep-0478.txt +++ b/pep-0478.txt @@ -27,7 +27,7 @@ - 3.5 Release Manager: Larry Hastings - Windows installers: Steve Dower -- Mac installers: Ned Deily / Ronald Oussoren +- Mac installers: Ned Deily - Documentation: Georg Brandl -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Mon Jan 12 21:45:53 2015 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 12 Jan 2015 20:45:53 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319777=3A_Provide_?= =?utf-8?q?a_home=28=29_classmethod_on_Path_objects=2E?= Message-ID: <20150112204538.22399.41892@psf.io> https://hg.python.org/cpython/rev/4a55b98314cd changeset: 94119:4a55b98314cd user: Antoine Pitrou date: Mon Jan 12 21:03:41 2015 +0100 summary: Issue #19777: Provide a home() classmethod on Path objects. Contributed by Victor Salgado and Mayank Tripathi. files: Doc/library/pathlib.rst | 11 +++++++++++ Lib/pathlib.py | 7 +++++++ Lib/test/test_pathlib.py | 11 +++++++++++ Misc/ACKS | 2 ++ Misc/NEWS | 3 +++ 5 files changed, 34 insertions(+), 0 deletions(-) diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -628,6 +628,17 @@ PosixPath('/home/antoine/pathlib') +.. classmethod:: Path.home() + + Return a new path object representing the user's home directory (as + returned by :func:`os.path.expanduser` with ``~`` construct):: + + >>> Path.home() + PosixPath('/home/antoine') + + .. versionadded:: 3.5 + + .. method:: Path.stat() Return information about this path (similarly to :func:`os.stat`). diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -1008,6 +1008,13 @@ """ return cls(os.getcwd()) + @classmethod + def home(cls): + """Return a new path pointing to the user's home directory (as + returned by os.path.expanduser('~')). + """ + return cls(cls()._flavour.gethomedir(None)) + def samefile(self, other_path): """Return whether `other_file` is the same or not as this file. (as returned by os.path.samefile(file, other_file)). diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1261,6 +1261,17 @@ p = self.cls.cwd() self._test_cwd(p) + def _test_home(self, p): + q = self.cls(os.path.expanduser('~')) + self.assertEqual(p, q) + self.assertEqual(str(p), str(q)) + self.assertIs(type(p), type(q)) + self.assertTrue(p.is_absolute()) + + def test_home(self): + p = self.cls.home() + self._test_home(p) + def test_samefile(self): fileA_path = os.path.join(BASE, 'fileA') fileB_path = os.path.join(BASE, 'dirB', 'fileB') diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1201,6 +1201,7 @@ Suman Saha Hajime Saitou George Sakkis +Victor Salgado Rich Salz Kevin Samborn Adrian Sampson @@ -1390,6 +1391,7 @@ Nathan Trapuzzano Laurence Tratt Alberto Trevino +Mayank Tripathi Matthias Troffaes Tom Tromey John Tromp diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,9 @@ Library ------- +- Issue #19777: Provide a home() classmethod on Path objects. Contributed + by Victor Salgado and Mayank Tripathi. + - Issue #23206: Make ``json.dumps(..., ensure_ascii=False)`` as fast as the default case of ``ensure_ascii=True``. Patch by Naoki Inada. -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Tue Jan 13 09:20:25 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 13 Jan 2015 09:20:25 +0100 Subject: [Python-checkins] Daily reference leaks (4a55b98314cd): sum=6 Message-ID: results for 4a55b98314cd on branch "default" -------------------------------------------- test_asyncio leaked [0, 3, 0] memory blocks, sum=3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogAZ_5D3', '-x'] From python-checkins at python.org Tue Jan 13 10:01:58 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 09:01:58 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323209=2C_=2323225=3A_selectors=2EBaseSelector?= =?utf-8?q?=2Eget=5Fkey=28=29_now_raises_a?= Message-ID: <20150113090148.8745.98932@psf.io> https://hg.python.org/cpython/rev/6e7403bc906f changeset: 94121:6e7403bc906f parent: 94119:4a55b98314cd parent: 94120:1544bdc409be user: Victor Stinner date: Tue Jan 13 10:00:55 2015 +0100 summary: Issue #23209, #23225: selectors.BaseSelector.get_key() now raises a RuntimeError if the selector is closed. And selectors.BaseSelector.close() now clears its internal reference to the selector mapping to break a reference cycle. Initial patch written by Martin Richard. files: Lib/selectors.py | 3 +++ Lib/test/test_selectors.py | 11 +++++++---- Misc/NEWS | 5 +++++ 3 files changed, 15 insertions(+), 4 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -174,6 +174,8 @@ SelectorKey for this file object """ mapping = self.get_map() + if mapping is None: + raise RuntimeError('Selector is closed') try: return mapping[fileobj] except KeyError: @@ -256,6 +258,7 @@ def close(self): self._fd_to_key.clear() + self._map = None def get_map(self): return self._map diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -178,14 +178,17 @@ s = self.SELECTOR() self.addCleanup(s.close) + mapping = s.get_map() rd, wr = self.make_socketpair() s.register(rd, selectors.EVENT_READ) s.register(wr, selectors.EVENT_WRITE) s.close() - self.assertRaises(KeyError, s.get_key, rd) - self.assertRaises(KeyError, s.get_key, wr) + self.assertRaises(RuntimeError, s.get_key, rd) + self.assertRaises(RuntimeError, s.get_key, wr) + self.assertRaises(KeyError, mapping.__getitem__, rd) + self.assertRaises(KeyError, mapping.__getitem__, wr) def test_get_key(self): s = self.SELECTOR() @@ -252,8 +255,8 @@ sel.register(rd, selectors.EVENT_READ) sel.register(wr, selectors.EVENT_WRITE) - self.assertRaises(KeyError, s.get_key, rd) - self.assertRaises(KeyError, s.get_key, wr) + self.assertRaises(RuntimeError, s.get_key, rd) + self.assertRaises(RuntimeError, s.get_key, wr) def test_fileno(self): s = self.SELECTOR() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,11 @@ Library ------- +- Issue #23209, #23225: selectors.BaseSelector.get_key() now raises a + RuntimeError if the selector is closed. And selectors.BaseSelector.close() + now clears its internal reference to the selector mapping to break a + reference cycle. Initial patch written by Martin Richard. + - Issue #19777: Provide a home() classmethod on Path objects. Contributed by Victor Salgado and Mayank Tripathi. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 13 10:01:58 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 09:01:58 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjA5?= =?utf-8?q?=2C_=2323225=3A_selectors=2EBaseSelector=2Eclose=28=29_now_clea?= =?utf-8?q?rs_its_internal?= Message-ID: <20150113090148.125888.67105@psf.io> https://hg.python.org/cpython/rev/1544bdc409be changeset: 94120:1544bdc409be branch: 3.4 parent: 94117:1b145e8ae4be user: Victor Stinner date: Tue Jan 13 09:58:33 2015 +0100 summary: Issue #23209, #23225: selectors.BaseSelector.close() now clears its internal reference to the selector mapping to break a reference cycle. Initial patch written by Martin Richard. files: Lib/selectors.py | 3 +++ Lib/test/test_selectors.py | 3 +++ Misc/NEWS | 4 ++++ 3 files changed, 10 insertions(+), 0 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -175,6 +175,8 @@ """ mapping = self.get_map() try: + if mapping is None: + raise KeyError return mapping[fileobj] except KeyError: raise KeyError("{!r} is not registered".format(fileobj)) from None @@ -256,6 +258,7 @@ def close(self): self._fd_to_key.clear() + self._map = None def get_map(self): return self._map diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -180,6 +180,7 @@ s = self.SELECTOR() self.addCleanup(s.close) + mapping = s.get_map() rd, wr = self.make_socketpair() s.register(rd, selectors.EVENT_READ) @@ -188,6 +189,8 @@ s.close() self.assertRaises(KeyError, s.get_key, rd) self.assertRaises(KeyError, s.get_key, wr) + self.assertRaises(KeyError, mapping.__getitem__, rd) + self.assertRaises(KeyError, mapping.__getitem__, wr) def test_get_key(self): s = self.SELECTOR() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,10 @@ Library ------- +- Issue #23209, #23225: selectors.BaseSelector.close() now clears its internal + reference to the selector mapping to break a reference cycle. Initial patch + written by Martin Richard. + - Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The availability of the function is checked during the compilation. Patch written by Bernard Spil. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 13 15:20:40 2015 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 13 Jan 2015 14:20:40 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_fix_instances_?= =?utf-8?q?of_consecutive_articles_=28closes_=2323221=29?= Message-ID: <20150113142039.22413.56688@psf.io> https://hg.python.org/cpython/rev/b168c41f2e3f changeset: 94122:b168c41f2e3f branch: 3.4 parent: 94120:1544bdc409be user: Benjamin Peterson date: Tue Jan 13 09:17:24 2015 -0500 summary: fix instances of consecutive articles (closes #23221) Patch by Karan Goel. files: Doc/c-api/exceptions.rst | 2 +- Doc/c-api/init.rst | 2 +- Doc/c-api/structures.rst | 2 +- Doc/distutils/apiref.rst | 2 +- Doc/library/unittest.mock.rst | 2 +- Include/dynamic_annotations.h | 2 +- Include/unicodeobject.h | 2 +- Lib/distutils/dir_util.py | 2 +- Lib/http/cookiejar.py | 2 +- Lib/lib2to3/fixes/fix_exitfunc.py | 2 +- Lib/socket.py | 2 +- Lib/test/test_argparse.py | 2 +- Misc/ACKS | 1 + Modules/_ctypes/_ctypes.c | 2 +- 14 files changed, 14 insertions(+), 13 deletions(-) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -64,7 +64,7 @@ Do not compare the return value to a specific exception; use :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could easily fail since the exception may be an instance instead of a class, in the - case of a class exception, or it may the a subclass of the expected exception.) + case of a class exception, or it may be a subclass of the expected exception.) .. c:function:: int PyErr_ExceptionMatches(PyObject *exc) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1182,7 +1182,7 @@ .. c:function:: PyThreadState * PyInterpreterState_ThreadHead(PyInterpreterState *interp) - Return the a pointer to the first :c:type:`PyThreadState` object in the list of + Return the pointer to the first :c:type:`PyThreadState` object in the list of threads associated with the interpreter *interp*. diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -131,7 +131,7 @@ types, but they always return :c:type:`PyObject\*`. If the function is not of the :c:type:`PyCFunction`, the compiler will require a cast in the method table. Even though :c:type:`PyCFunction` defines the first parameter as -:c:type:`PyObject\*`, it is common that the method implementation uses a the +:c:type:`PyObject\*`, it is common that the method implementation uses the specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -964,7 +964,7 @@ .. function:: create_tree(base_dir, files[, mode=0o777, verbose=0, dry_run=0]) Create all the empty directories under *base_dir* needed to put *files* there. - *base_dir* is just the a name of a directory which doesn't necessarily exist + *base_dir* is just the name of a directory which doesn't necessarily exist yet; *files* is a list of filenames to be interpreted relative to *base_dir*. *base_dir* + the directory portion of every file in *files* will be created if it doesn't already exist. *mode*, *verbose* and *dry_run* flags are as for diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -1499,7 +1499,7 @@ However, consider the alternative scenario where instead of ``from a import SomeClass`` module b does ``import a`` and ``some_function`` uses ``a.SomeClass``. Both of these import forms are common. In this case the class we want to patch is -being looked up on the a module and so we have to patch ``a.SomeClass`` instead:: +being looked up in the module and so we have to patch ``a.SomeClass`` instead:: @patch('a.SomeClass') diff --git a/Include/dynamic_annotations.h b/Include/dynamic_annotations.h --- a/Include/dynamic_annotations.h +++ b/Include/dynamic_annotations.h @@ -150,7 +150,7 @@ /* Report that a new memory at "address" of size "size" has been allocated. This might be used when the memory has been retrieved from a free list and - is about to be reused, or when a the locking discipline for a variable + is about to be reused, or when the locking discipline for a variable changes. */ #define _Py_ANNOTATE_NEW_MEMORY(address, size) \ AnnotateNewMemory(__FILE__, __LINE__, address, size) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -605,7 +605,7 @@ ); #endif -/* Initializes the canonical string representation from a the deprecated +/* Initializes the canonical string representation from the deprecated wstr/Py_UNICODE representation. This function is used to convert Unicode objects which were created using the old API to the new flexible format introduced with PEP 393. diff --git a/Lib/distutils/dir_util.py b/Lib/distutils/dir_util.py --- a/Lib/distutils/dir_util.py +++ b/Lib/distutils/dir_util.py @@ -81,7 +81,7 @@ """Create all the empty directories under 'base_dir' needed to put 'files' there. - 'base_dir' is just the a name of a directory which doesn't necessarily + 'base_dir' is just the name of a directory which doesn't necessarily exist yet; 'files' is a list of filenames to be interpreted relative to 'base_dir'. 'base_dir' + the directory portion of every file in 'files' will be created if it doesn't already exist. 'mode', 'verbose' and diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -1792,7 +1792,7 @@ def lwp_cookie_str(cookie): - """Return string representation of Cookie in an the LWP cookie file format. + """Return string representation of Cookie in the LWP cookie file format. Actually, the format is extended a bit -- see module docstring. diff --git a/Lib/lib2to3/fixes/fix_exitfunc.py b/Lib/lib2to3/fixes/fix_exitfunc.py --- a/Lib/lib2to3/fixes/fix_exitfunc.py +++ b/Lib/lib2to3/fixes/fix_exitfunc.py @@ -35,7 +35,7 @@ self.sys_import = None def transform(self, node, results): - # First, find a the sys import. We'll just hope it's global scope. + # First, find the sys import. We'll just hope it's global scope. if "sys_import" in results: if self.sys_import is None: self.sys_import = results["sys_import"] diff --git a/Lib/socket.py b/Lib/socket.py --- a/Lib/socket.py +++ b/Lib/socket.py @@ -299,7 +299,7 @@ def fromshare(info): """ fromshare(info) -> socket object - Create a socket object from a the bytes object returned by + Create a socket object from the bytes object returned by socket.share(pid). """ return socket(0, 0, 0, info) diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -644,7 +644,7 @@ class TestOptionalsRequired(ParserTestCase): - """Tests the an optional action that is required""" + """Tests an optional action that is required""" argument_signatures = [ Sig('-x', type=int, required=True), diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -483,6 +483,7 @@ Matt Giuca Wim Glenn Michael Goderbauer +Karan Goel Jeroen Van Goey Christoph Gohlke Tim Golden diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -4830,7 +4830,7 @@ *(void **)self->b_ptr = dst->b_ptr; /* - A Pointer instance must keep a the value it points to alive. So, a + A Pointer instance must keep the value it points to alive. So, a pointer instance has b_length set to 2 instead of 1, and we set 'value' itself as the second item of the b_objects list, additionally. */ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 13 15:20:39 2015 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 13 Jan 2015 14:20:39 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_fix_instances_?= =?utf-8?q?of_consecutive_articles_=28closes_=2323221=29?= Message-ID: <20150113142039.11573.19012@psf.io> https://hg.python.org/cpython/rev/6a19e37ce94d changeset: 94123:6a19e37ce94d branch: 2.7 parent: 94116:ad92f9b2fe04 user: Benjamin Peterson date: Tue Jan 13 09:17:24 2015 -0500 summary: fix instances of consecutive articles (closes #23221) Patch by Karan Goel. files: Doc/c-api/exceptions.rst | 2 +- Doc/c-api/init.rst | 2 +- Doc/c-api/structures.rst | 2 +- Doc/distutils/apiref.rst | 2 +- Lib/_LWPCookieJar.py | 2 +- Lib/distutils/dir_util.py | 2 +- Lib/lib2to3/fixes/fix_exitfunc.py | 2 +- Lib/test/test_argparse.py | 2 +- Misc/ACKS | 1 + Modules/_ctypes/_ctypes.c | 2 +- 10 files changed, 10 insertions(+), 9 deletions(-) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -70,7 +70,7 @@ Do not compare the return value to a specific exception; use :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could easily fail since the exception may be an instance instead of a class, in the - case of a class exception, or it may the a subclass of the expected exception.) + case of a class exception, or it may be a subclass of the expected exception.) .. c:function:: int PyErr_ExceptionMatches(PyObject *exc) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1136,7 +1136,7 @@ .. c:function:: PyThreadState * PyInterpreterState_ThreadHead(PyInterpreterState *interp) - Return the a pointer to the first :c:type:`PyThreadState` object in the list of + Return the pointer to the first :c:type:`PyThreadState` object in the list of threads associated with the interpreter *interp*. .. versionadded:: 2.2 diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -122,7 +122,7 @@ types, but they always return :c:type:`PyObject\*`. If the function is not of the :c:type:`PyCFunction`, the compiler will require a cast in the method table. Even though :c:type:`PyCFunction` defines the first parameter as -:c:type:`PyObject\*`, it is common that the method implementation uses a the +:c:type:`PyObject\*`, it is common that the method implementation uses the specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -970,7 +970,7 @@ .. function:: create_tree(base_dir, files[, mode=0777, verbose=0, dry_run=0]) Create all the empty directories under *base_dir* needed to put *files* there. - *base_dir* is just the a name of a directory which doesn't necessarily exist + *base_dir* is just the name of a directory which doesn't necessarily exist yet; *files* is a list of filenames to be interpreted relative to *base_dir*. *base_dir* + the directory portion of every file in *files* will be created if it doesn't already exist. *mode*, *verbose* and *dry_run* flags are as for diff --git a/Lib/_LWPCookieJar.py b/Lib/_LWPCookieJar.py --- a/Lib/_LWPCookieJar.py +++ b/Lib/_LWPCookieJar.py @@ -18,7 +18,7 @@ iso2time, time2isoz) def lwp_cookie_str(cookie): - """Return string representation of Cookie in an the LWP cookie file format. + """Return string representation of Cookie in the LWP cookie file format. Actually, the format is extended a bit -- see module docstring. diff --git a/Lib/distutils/dir_util.py b/Lib/distutils/dir_util.py --- a/Lib/distutils/dir_util.py +++ b/Lib/distutils/dir_util.py @@ -83,7 +83,7 @@ """Create all the empty directories under 'base_dir' needed to put 'files' there. - 'base_dir' is just the a name of a directory which doesn't necessarily + 'base_dir' is just the name of a directory which doesn't necessarily exist yet; 'files' is a list of filenames to be interpreted relative to 'base_dir'. 'base_dir' + the directory portion of every file in 'files' will be created if it doesn't already exist. 'mode', 'verbose' and diff --git a/Lib/lib2to3/fixes/fix_exitfunc.py b/Lib/lib2to3/fixes/fix_exitfunc.py --- a/Lib/lib2to3/fixes/fix_exitfunc.py +++ b/Lib/lib2to3/fixes/fix_exitfunc.py @@ -35,7 +35,7 @@ self.sys_import = None def transform(self, node, results): - # First, find a the sys import. We'll just hope it's global scope. + # First, find the sys import. We'll just hope it's global scope. if "sys_import" in results: if self.sys_import is None: self.sys_import = results["sys_import"] diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -646,7 +646,7 @@ class TestOptionalsRequired(ParserTestCase): - """Tests the an optional action that is required""" + """Tests an optional action that is required""" argument_signatures = [ Sig('-x', type=int, required=True), diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -475,6 +475,7 @@ Matt Giuca Wim Glenn Michael Goderbauer +Karan Goel Jeroen Van Goey Christoph Gohlke Tim Golden diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -5060,7 +5060,7 @@ *(void **)self->b_ptr = dst->b_ptr; /* - A Pointer instance must keep a the value it points to alive. So, a + A Pointer instance must keep the value it points to alive. So, a pointer instance has b_length set to 2 instead of 1, and we set 'value' itself as the second item of the b_objects list, additionally. */ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 13 15:20:39 2015 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 13 Jan 2015 14:20:39 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMyMjEp?= Message-ID: <20150113142039.125906.62520@psf.io> https://hg.python.org/cpython/rev/c16b76c7c8ba changeset: 94124:c16b76c7c8ba parent: 94121:6e7403bc906f parent: 94122:b168c41f2e3f user: Benjamin Peterson date: Tue Jan 13 09:20:31 2015 -0500 summary: merge 3.4 (#23221) files: Doc/c-api/exceptions.rst | 2 +- Doc/c-api/init.rst | 2 +- Doc/c-api/structures.rst | 2 +- Doc/distutils/apiref.rst | 2 +- Doc/library/unittest.mock.rst | 2 +- Include/dynamic_annotations.h | 2 +- Include/unicodeobject.h | 2 +- Lib/distutils/dir_util.py | 2 +- Lib/http/cookiejar.py | 2 +- Lib/lib2to3/fixes/fix_exitfunc.py | 2 +- Lib/socket.py | 2 +- Lib/test/test_argparse.py | 2 +- Misc/ACKS | 1 + Modules/_ctypes/_ctypes.c | 2 +- 14 files changed, 14 insertions(+), 13 deletions(-) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -350,7 +350,7 @@ Do not compare the return value to a specific exception; use :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could easily fail since the exception may be an instance instead of a class, in the - case of a class exception, or it may the a subclass of the expected exception.) + case of a class exception, or it may be a subclass of the expected exception.) .. c:function:: int PyErr_ExceptionMatches(PyObject *exc) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1197,7 +1197,7 @@ .. c:function:: PyThreadState * PyInterpreterState_ThreadHead(PyInterpreterState *interp) - Return the a pointer to the first :c:type:`PyThreadState` object in the list of + Return the pointer to the first :c:type:`PyThreadState` object in the list of threads associated with the interpreter *interp*. diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -131,7 +131,7 @@ types, but they always return :c:type:`PyObject\*`. If the function is not of the :c:type:`PyCFunction`, the compiler will require a cast in the method table. Even though :c:type:`PyCFunction` defines the first parameter as -:c:type:`PyObject\*`, it is common that the method implementation uses a the +:c:type:`PyObject\*`, it is common that the method implementation uses the specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -964,7 +964,7 @@ .. function:: create_tree(base_dir, files[, mode=0o777, verbose=0, dry_run=0]) Create all the empty directories under *base_dir* needed to put *files* there. - *base_dir* is just the a name of a directory which doesn't necessarily exist + *base_dir* is just the name of a directory which doesn't necessarily exist yet; *files* is a list of filenames to be interpreted relative to *base_dir*. *base_dir* + the directory portion of every file in *files* will be created if it doesn't already exist. *mode*, *verbose* and *dry_run* flags are as for diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -1541,7 +1541,7 @@ However, consider the alternative scenario where instead of ``from a import SomeClass`` module b does ``import a`` and ``some_function`` uses ``a.SomeClass``. Both of these import forms are common. In this case the class we want to patch is -being looked up on the a module and so we have to patch ``a.SomeClass`` instead:: +being looked up in the module and so we have to patch ``a.SomeClass`` instead:: @patch('a.SomeClass') diff --git a/Include/dynamic_annotations.h b/Include/dynamic_annotations.h --- a/Include/dynamic_annotations.h +++ b/Include/dynamic_annotations.h @@ -150,7 +150,7 @@ /* Report that a new memory at "address" of size "size" has been allocated. This might be used when the memory has been retrieved from a free list and - is about to be reused, or when a the locking discipline for a variable + is about to be reused, or when the locking discipline for a variable changes. */ #define _Py_ANNOTATE_NEW_MEMORY(address, size) \ AnnotateNewMemory(__FILE__, __LINE__, address, size) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -605,7 +605,7 @@ ); #endif -/* Initializes the canonical string representation from a the deprecated +/* Initializes the canonical string representation from the deprecated wstr/Py_UNICODE representation. This function is used to convert Unicode objects which were created using the old API to the new flexible format introduced with PEP 393. diff --git a/Lib/distutils/dir_util.py b/Lib/distutils/dir_util.py --- a/Lib/distutils/dir_util.py +++ b/Lib/distutils/dir_util.py @@ -81,7 +81,7 @@ """Create all the empty directories under 'base_dir' needed to put 'files' there. - 'base_dir' is just the a name of a directory which doesn't necessarily + 'base_dir' is just the name of a directory which doesn't necessarily exist yet; 'files' is a list of filenames to be interpreted relative to 'base_dir'. 'base_dir' + the directory portion of every file in 'files' will be created if it doesn't already exist. 'mode', 'verbose' and diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -1792,7 +1792,7 @@ def lwp_cookie_str(cookie): - """Return string representation of Cookie in an the LWP cookie file format. + """Return string representation of Cookie in the LWP cookie file format. Actually, the format is extended a bit -- see module docstring. diff --git a/Lib/lib2to3/fixes/fix_exitfunc.py b/Lib/lib2to3/fixes/fix_exitfunc.py --- a/Lib/lib2to3/fixes/fix_exitfunc.py +++ b/Lib/lib2to3/fixes/fix_exitfunc.py @@ -35,7 +35,7 @@ self.sys_import = None def transform(self, node, results): - # First, find a the sys import. We'll just hope it's global scope. + # First, find the sys import. We'll just hope it's global scope. if "sys_import" in results: if self.sys_import is None: self.sys_import = results["sys_import"] diff --git a/Lib/socket.py b/Lib/socket.py --- a/Lib/socket.py +++ b/Lib/socket.py @@ -450,7 +450,7 @@ def fromshare(info): """ fromshare(info) -> socket object - Create a socket object from a the bytes object returned by + Create a socket object from the bytes object returned by socket.share(pid). """ return socket(0, 0, 0, info) diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -635,7 +635,7 @@ class TestOptionalsRequired(ParserTestCase): - """Tests the an optional action that is required""" + """Tests an optional action that is required""" argument_signatures = [ Sig('-x', type=int, required=True), diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -488,6 +488,7 @@ Matt Giuca Wim Glenn Michael Goderbauer +Karan Goel Jeroen Van Goey Christoph Gohlke Tim Golden diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -4830,7 +4830,7 @@ *(void **)self->b_ptr = dst->b_ptr; /* - A Pointer instance must keep a the value it points to alive. So, a + A Pointer instance must keep the value it points to alive. So, a pointer instance has b_length set to 2 instead of 1, and we set 'value' itself as the second item of the b_objects list, additionally. */ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 13 16:15:12 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 15:15:12 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Tulip_issue_18?= =?utf-8?q?4=3A_Fix_test=5Fpipe=28=29_on_Windows?= Message-ID: <20150113151507.8757.23143@psf.io> https://hg.python.org/cpython/rev/0387862a5675 changeset: 94126:0387862a5675 branch: 3.4 user: Victor Stinner date: Tue Jan 13 16:13:06 2015 +0100 summary: Tulip issue 184: Fix test_pipe() on Windows Pass explicitly the event loop to StreamReaderProtocol. files: Lib/test/test_asyncio/test_windows_events.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -67,7 +67,8 @@ clients = [] for i in range(5): stream_reader = asyncio.StreamReader(loop=self.loop) - protocol = asyncio.StreamReaderProtocol(stream_reader) + protocol = asyncio.StreamReaderProtocol(stream_reader, + loop=self.loop) trans, proto = yield from self.loop.create_pipe_connection( lambda: protocol, ADDRESS) self.assertIsInstance(trans, asyncio.Transport) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 13 16:15:12 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 15:15:12 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150113151507.11589.71651@psf.io> https://hg.python.org/cpython/rev/56f717235c45 changeset: 94127:56f717235c45 parent: 94124:c16b76c7c8ba parent: 94126:0387862a5675 user: Victor Stinner date: Tue Jan 13 16:13:36 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/proactor_events.py | 8 +++++++- Lib/test/test_asyncio/test_windows_events.py | 3 ++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -387,13 +387,19 @@ raise RuntimeError("Cannot close a running event loop") if self.is_closed(): return + + # Call these methods before closing the event loop (before calling + # BaseEventLoop.close), because they can schedule callbacks with + # call_soon(), which is forbidden when the event loop is closed. self._stop_accept_futures() self._close_self_pipe() - super().close() self._proactor.close() self._proactor = None self._selector = None + # Close the event loop + super().close() + def sock_recv(self, sock, n): return self._proactor.recv(sock, n) diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -67,7 +67,8 @@ clients = [] for i in range(5): stream_reader = asyncio.StreamReader(loop=self.loop) - protocol = asyncio.StreamReaderProtocol(stream_reader) + protocol = asyncio.StreamReaderProtocol(stream_reader, + loop=self.loop) trans, proto = yield from self.loop.create_pipe_connection( lambda: protocol, ADDRESS) self.assertIsInstance(trans, asyncio.Transport) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 13 16:15:12 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 15:15:12 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyOTIy?= =?utf-8?q?=3A_Fix_ProactorEventLoop=2Eclose=28=29?= Message-ID: <20150113151506.8757.25307@psf.io> https://hg.python.org/cpython/rev/6c473f82309d changeset: 94125:6c473f82309d branch: 3.4 parent: 94122:b168c41f2e3f user: Victor Stinner date: Tue Jan 13 16:11:19 2015 +0100 summary: Issue #22922: Fix ProactorEventLoop.close() Close the IocpProactor before closing the event loop. IocpProactor.close() can call loop.call_soon(), which is forbidden when the event loop is closed. files: Lib/asyncio/proactor_events.py | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -387,13 +387,19 @@ raise RuntimeError("Cannot close a running event loop") if self.is_closed(): return + + # Call these methods before closing the event loop (before calling + # BaseEventLoop.close), because they can schedule callbacks with + # call_soon(), which is forbidden when the event loop is closed. self._stop_accept_futures() self._close_self_pipe() - super().close() self._proactor.close() self._proactor = None self._selector = None + # Close the event loop + super().close() + def sock_recv(self, sock, n): return self._proactor.recv(sock, n) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 00:21:43 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 23:21:43 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=3A_new_SSL_implementation=29?= Message-ID: <20150113232143.11587.50003@psf.io> https://hg.python.org/cpython/rev/dfe532dd0138 changeset: 94129:dfe532dd0138 parent: 94127:56f717235c45 parent: 94128:432b817611f2 user: Victor Stinner date: Wed Jan 14 00:19:55 2015 +0100 summary: Merge 3.4 (asyncio: new SSL implementation) files: Lib/asyncio/proactor_events.py | 31 +- Lib/asyncio/selector_events.py | 45 +- Lib/asyncio/sslproto.py | 640 ++++++++++ Lib/asyncio/test_utils.py | 5 + Lib/test/test_asyncio/test_events.py | 66 +- Lib/test/test_asyncio/test_selector_events.py | 6 +- 6 files changed, 751 insertions(+), 42 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -11,6 +11,7 @@ from . import base_events from . import constants from . import futures +from . import sslproto from . import transports from .log import logger @@ -367,6 +368,20 @@ return _ProactorSocketTransport(self, sock, protocol, waiter, extra, server) + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, + *, server_side=False, server_hostname=None, + extra=None, server=None): + if not sslproto._is_sslproto_available(): + raise NotImplementedError("Proactor event loop requires Python 3.5" + " or newer (ssl.MemoryBIO) to support " + "SSL") + + ssl_protocol = sslproto.SSLProtocol(self, protocol, sslcontext, waiter, + server_side, server_hostname) + _ProactorSocketTransport(self, rawsock, ssl_protocol, + extra=extra, server=server) + return ssl_protocol._app_transport + def _make_duplex_pipe_transport(self, sock, protocol, waiter=None, extra=None): return _ProactorDuplexPipeTransport(self, @@ -455,9 +470,8 @@ def _write_to_self(self): self._csock.send(b'\0') - def _start_serving(self, protocol_factory, sock, ssl=None, server=None): - if ssl: - raise ValueError('IocpEventLoop is incompatible with SSL.') + def _start_serving(self, protocol_factory, sock, + sslcontext=None, server=None): def loop(f=None): try: @@ -467,9 +481,14 @@ logger.debug("%r got a new connection from %r: %r", server, addr, conn) protocol = protocol_factory() - self._make_socket_transport( - conn, protocol, - extra={'peername': addr}, server=server) + if sslcontext is not None: + self._make_ssl_transport( + conn, protocol, sslcontext, server_side=True, + extra={'peername': addr}, server=server) + else: + self._make_socket_transport( + conn, protocol, + extra={'peername': addr}, server=server) if self.is_closed(): return f = self._proactor.accept(sock) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -10,6 +10,7 @@ import errno import functools import socket +import sys try: import ssl except ImportError: # pragma: no cover @@ -21,6 +22,7 @@ from . import futures from . import selectors from . import transports +from . import sslproto from .log import logger @@ -58,6 +60,24 @@ def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, server_side=False, server_hostname=None, extra=None, server=None): + if not sslproto._is_sslproto_available(): + return self._make_legacy_ssl_transport( + rawsock, protocol, sslcontext, waiter, + server_side=server_side, server_hostname=server_hostname, + extra=extra, server=server) + + ssl_protocol = sslproto.SSLProtocol(self, protocol, sslcontext, waiter, + server_side, server_hostname) + _SelectorSocketTransport(self, rawsock, ssl_protocol, + extra=extra, server=server) + return ssl_protocol._app_transport + + def _make_legacy_ssl_transport(self, rawsock, protocol, sslcontext, + waiter, *, + server_side=False, server_hostname=None, + extra=None, server=None): + # Use the legacy API: SSL_write, SSL_read, etc. The legacy API is used + # on Python 3.4 and older, when ssl.MemoryBIO is not available. return _SelectorSslTransport( self, rawsock, protocol, sslcontext, waiter, server_side, server_hostname, extra, server) @@ -508,7 +528,8 @@ def _fatal_error(self, exc, message='Fatal error on transport'): # Should be called from exception handler only. - if isinstance(exc, (BrokenPipeError, ConnectionResetError)): + if isinstance(exc, (BrokenPipeError, + ConnectionResetError, ConnectionAbortedError)): if self._loop.get_debug(): logger.debug("%r: %s", self, message, exc_info=True) else: @@ -683,26 +704,8 @@ if ssl is None: raise RuntimeError('stdlib ssl module not available') - if server_side: - if not sslcontext: - raise ValueError('Server side ssl needs a valid SSLContext') - else: - if not sslcontext: - # Client side may pass ssl=True to use a default - # context; in that case the sslcontext passed is None. - # The default is secure for client connections. - if hasattr(ssl, 'create_default_context'): - # Python 3.4+: use up-to-date strong settings. - sslcontext = ssl.create_default_context() - if not server_hostname: - sslcontext.check_hostname = False - else: - # Fallback for Python 3.3. - sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - sslcontext.options |= ssl.OP_NO_SSLv2 - sslcontext.options |= ssl.OP_NO_SSLv3 - sslcontext.set_default_verify_paths() - sslcontext.verify_mode = ssl.CERT_REQUIRED + if not sslcontext: + sslcontext = sslproto._create_transport_context(server_side, server_hostname) wrap_kwargs = { 'server_side': server_side, diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py new file mode 100644 --- /dev/null +++ b/Lib/asyncio/sslproto.py @@ -0,0 +1,640 @@ +import collections +try: + import ssl +except ImportError: # pragma: no cover + ssl = None + +from . import protocols +from . import transports +from .log import logger + + +def _create_transport_context(server_side, server_hostname): + if server_side: + raise ValueError('Server side SSL needs a valid SSLContext') + + # Client side may pass ssl=True to use a default + # context; in that case the sslcontext passed is None. + # The default is secure for client connections. + if hasattr(ssl, 'create_default_context'): + # Python 3.4+: use up-to-date strong settings. + sslcontext = ssl.create_default_context() + if not server_hostname: + sslcontext.check_hostname = False + else: + # Fallback for Python 3.3. + sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + sslcontext.options |= ssl.OP_NO_SSLv2 + sslcontext.options |= ssl.OP_NO_SSLv3 + sslcontext.set_default_verify_paths() + sslcontext.verify_mode = ssl.CERT_REQUIRED + return sslcontext + + +def _is_sslproto_available(): + return hasattr(ssl, "MemoryBIO") + + +# States of an _SSLPipe. +_UNWRAPPED = "UNWRAPPED" +_DO_HANDSHAKE = "DO_HANDSHAKE" +_WRAPPED = "WRAPPED" +_SHUTDOWN = "SHUTDOWN" + + +class _SSLPipe(object): + """An SSL "Pipe". + + An SSL pipe allows you to communicate with an SSL/TLS protocol instance + through memory buffers. It can be used to implement a security layer for an + existing connection where you don't have access to the connection's file + descriptor, or for some reason you don't want to use it. + + An SSL pipe can be in "wrapped" and "unwrapped" mode. In unwrapped mode, + data is passed through untransformed. In wrapped mode, application level + data is encrypted to SSL record level data and vice versa. The SSL record + level is the lowest level in the SSL protocol suite and is what travels + as-is over the wire. + + An SslPipe initially is in "unwrapped" mode. To start SSL, call + do_handshake(). To shutdown SSL again, call unwrap(). + """ + + max_size = 256 * 1024 # Buffer size passed to read() + + def __init__(self, context, server_side, server_hostname=None): + """ + The *context* argument specifies the ssl.SSLContext to use. + + The *server_side* argument indicates whether this is a server side or + client side transport. + + The optional *server_hostname* argument can be used to specify the + hostname you are connecting to. You may only specify this parameter if + the _ssl module supports Server Name Indication (SNI). + """ + self._context = context + self._server_side = server_side + self._server_hostname = server_hostname + self._state = _UNWRAPPED + self._incoming = ssl.MemoryBIO() + self._outgoing = ssl.MemoryBIO() + self._sslobj = None + self._need_ssldata = False + self._handshake_cb = None + self._shutdown_cb = None + + @property + def context(self): + """The SSL context passed to the constructor.""" + return self._context + + @property + def ssl_object(self): + """The internal ssl.SSLObject instance. + + Return None if the pipe is not wrapped. + """ + return self._sslobj + + @property + def need_ssldata(self): + """Whether more record level data is needed to complete a handshake + that is currently in progress.""" + return self._need_ssldata + + @property + def wrapped(self): + """ + Whether a security layer is currently in effect. + + Return False during handshake. + """ + return self._state == _WRAPPED + + def do_handshake(self, callback=None): + """Start the SSL handshake. + + Return a list of ssldata. A ssldata element is a list of buffers + + The optional *callback* argument can be used to install a callback that + will be called when the handshake is complete. The callback will be + called with None if successful, else an exception instance. + """ + if self._state != _UNWRAPPED: + raise RuntimeError('handshake in progress or completed') + self._sslobj = self._context.wrap_bio( + self._incoming, self._outgoing, + server_side=self._server_side, + server_hostname=self._server_hostname) + self._state = _DO_HANDSHAKE + self._handshake_cb = callback + ssldata, appdata = self.feed_ssldata(b'', only_handshake=True) + assert len(appdata) == 0 + return ssldata + + def shutdown(self, callback=None): + """Start the SSL shutdown sequence. + + Return a list of ssldata. A ssldata element is a list of buffers + + The optional *callback* argument can be used to install a callback that + will be called when the shutdown is complete. The callback will be + called without arguments. + """ + if self._state == _UNWRAPPED: + raise RuntimeError('no security layer present') + if self._state == _SHUTDOWN: + raise RuntimeError('shutdown in progress') + assert self._state in (_WRAPPED, _DO_HANDSHAKE) + self._state = _SHUTDOWN + self._shutdown_cb = callback + ssldata, appdata = self.feed_ssldata(b'') + assert appdata == [] or appdata == [b''] + return ssldata + + def feed_eof(self): + """Send a potentially "ragged" EOF. + + This method will raise an SSL_ERROR_EOF exception if the EOF is + unexpected. + """ + self._incoming.write_eof() + ssldata, appdata = self.feed_ssldata(b'') + assert appdata == [] or appdata == [b''] + + def feed_ssldata(self, data, only_handshake=False): + """Feed SSL record level data into the pipe. + + The data must be a bytes instance. It is OK to send an empty bytes + instance. This can be used to get ssldata for a handshake initiated by + this endpoint. + + Return a (ssldata, appdata) tuple. The ssldata element is a list of + buffers containing SSL data that needs to be sent to the remote SSL. + + The appdata element is a list of buffers containing plaintext data that + needs to be forwarded to the application. The appdata list may contain + an empty buffer indicating an SSL "close_notify" alert. This alert must + be acknowledged by calling shutdown(). + """ + if self._state == _UNWRAPPED: + # If unwrapped, pass plaintext data straight through. + if data: + appdata = [data] + else: + appdata = [] + return ([], appdata) + + self._need_ssldata = False + if data: + self._incoming.write(data) + + ssldata = [] + appdata = [] + try: + if self._state == _DO_HANDSHAKE: + # Call do_handshake() until it doesn't raise anymore. + self._sslobj.do_handshake() + self._state = _WRAPPED + if self._handshake_cb: + self._handshake_cb(None) + if only_handshake: + return (ssldata, appdata) + # Handshake done: execute the wrapped block + + if self._state == _WRAPPED: + # Main state: read data from SSL until close_notify + while True: + chunk = self._sslobj.read(self.max_size) + appdata.append(chunk) + if not chunk: # close_notify + break + + elif self._state == _SHUTDOWN: + # Call shutdown() until it doesn't raise anymore. + self._sslobj.unwrap() + self._sslobj = None + self._state = _UNWRAPPED + if self._shutdown_cb: + self._shutdown_cb() + + elif self._state == _UNWRAPPED: + # Drain possible plaintext data after close_notify. + appdata.append(self._incoming.read()) + except (ssl.SSLError, ssl.CertificateError) as exc: + if getattr(exc, 'errno', None) not in ( + ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE, + ssl.SSL_ERROR_SYSCALL): + if self._state == _DO_HANDSHAKE and self._handshake_cb: + self._handshake_cb(exc) + raise + self._need_ssldata = (exc.errno == ssl.SSL_ERROR_WANT_READ) + + # Check for record level data that needs to be sent back. + # Happens for the initial handshake and renegotiations. + if self._outgoing.pending: + ssldata.append(self._outgoing.read()) + return (ssldata, appdata) + + def feed_appdata(self, data, offset=0): + """Feed plaintext data into the pipe. + + Return an (ssldata, offset) tuple. The ssldata element is a list of + buffers containing record level data that needs to be sent to the + remote SSL instance. The offset is the number of plaintext bytes that + were processed, which may be less than the length of data. + + NOTE: In case of short writes, this call MUST be retried with the SAME + buffer passed into the *data* argument (i.e. the id() must be the + same). This is an OpenSSL requirement. A further particularity is that + a short write will always have offset == 0, because the _ssl module + does not enable partial writes. And even though the offset is zero, + there will still be encrypted data in ssldata. + """ + assert 0 <= offset <= len(data) + if self._state == _UNWRAPPED: + # pass through data in unwrapped mode + if offset < len(data): + ssldata = [data[offset:]] + else: + ssldata = [] + return (ssldata, len(data)) + + ssldata = [] + view = memoryview(data) + while True: + self._need_ssldata = False + try: + if offset < len(view): + offset += self._sslobj.write(view[offset:]) + except ssl.SSLError as exc: + # It is not allowed to call write() after unwrap() until the + # close_notify is acknowledged. We return the condition to the + # caller as a short write. + if exc.reason == 'PROTOCOL_IS_SHUTDOWN': + exc.errno = ssl.SSL_ERROR_WANT_READ + if exc.errno not in (ssl.SSL_ERROR_WANT_READ, + ssl.SSL_ERROR_WANT_WRITE, + ssl.SSL_ERROR_SYSCALL): + raise + self._need_ssldata = (exc.errno == ssl.SSL_ERROR_WANT_READ) + + # See if there's any record level data back for us. + if self._outgoing.pending: + ssldata.append(self._outgoing.read()) + if offset == len(view) or self._need_ssldata: + break + return (ssldata, offset) + + +class _SSLProtocolTransport(transports._FlowControlMixin, + transports.Transport): + + def __init__(self, loop, ssl_protocol, app_protocol): + self._loop = loop + self._ssl_protocol = ssl_protocol + self._app_protocol = app_protocol + + def get_extra_info(self, name, default=None): + """Get optional transport information.""" + return self._ssl_protocol._get_extra_info(name, default) + + def close(self): + """Close the transport. + + Buffered data will be flushed asynchronously. No more data + will be received. After all buffered data is flushed, the + protocol's connection_lost() method will (eventually) called + with None as its argument. + """ + self._ssl_protocol._start_shutdown() + + def pause_reading(self): + """Pause the receiving end. + + No data will be passed to the protocol's data_received() + method until resume_reading() is called. + """ + self._ssl_protocol._transport.pause_reading() + + def resume_reading(self): + """Resume the receiving end. + + Data received will once again be passed to the protocol's + data_received() method. + """ + self._ssl_protocol._transport.resume_reading() + + def set_write_buffer_limits(self, high=None, low=None): + """Set the high- and low-water limits for write flow control. + + These two values control when to call the protocol's + pause_writing() and resume_writing() methods. If specified, + the low-water limit must be less than or equal to the + high-water limit. Neither value can be negative. + + The defaults are implementation-specific. If only the + high-water limit is given, the low-water limit defaults to a + implementation-specific value less than or equal to the + high-water limit. Setting high to zero forces low to zero as + well, and causes pause_writing() to be called whenever the + buffer becomes non-empty. Setting low to zero causes + resume_writing() to be called only once the buffer is empty. + Use of zero for either limit is generally sub-optimal as it + reduces opportunities for doing I/O and computation + concurrently. + """ + self._ssl_protocol._transport.set_write_buffer_limits(high, low) + + def get_write_buffer_size(self): + """Return the current size of the write buffer.""" + return self._ssl_protocol._transport.get_write_buffer_size() + + def write(self, data): + """Write some data bytes to the transport. + + This does not block; it buffers the data and arranges for it + to be sent out asynchronously. + """ + if not isinstance(data, (bytes, bytearray, memoryview)): + raise TypeError("data: expecting a bytes-like instance, got {!r}" + .format(type(data).__name__)) + if not data: + return + self._ssl_protocol._write_appdata(data) + + def can_write_eof(self): + """Return True if this transport supports write_eof(), False if not.""" + return False + + def abort(self): + """Close the transport immediately. + + Buffered data will be lost. No more data will be received. + The protocol's connection_lost() method will (eventually) be + called with None as its argument. + """ + self._ssl_protocol._abort() + + +class SSLProtocol(protocols.Protocol): + """SSL protocol. + + Implementation of SSL on top of a socket using incoming and outgoing + buffers which are ssl.MemoryBIO objects. + """ + + def __init__(self, loop, app_protocol, sslcontext, waiter, + server_side=False, server_hostname=None): + if ssl is None: + raise RuntimeError('stdlib ssl module not available') + + if not sslcontext: + sslcontext = _create_transport_context(server_side, server_hostname) + + self._server_side = server_side + if server_hostname and not server_side: + self._server_hostname = server_hostname + else: + self._server_hostname = None + self._sslcontext = sslcontext + # SSL-specific extra info. More info are set when the handshake + # completes. + self._extra = dict(sslcontext=sslcontext) + + # App data write buffering + self._write_backlog = collections.deque() + self._write_buffer_size = 0 + + self._waiter = waiter + self._closing = False + self._loop = loop + self._app_protocol = app_protocol + self._app_transport = _SSLProtocolTransport(self._loop, + self, self._app_protocol) + self._sslpipe = None + self._session_established = False + self._in_handshake = False + self._in_shutdown = False + + def connection_made(self, transport): + """Called when the low-level connection is made. + + Start the SSL handshake. + """ + self._transport = transport + self._sslpipe = _SSLPipe(self._sslcontext, + self._server_side, + self._server_hostname) + self._start_handshake() + + def connection_lost(self, exc): + """Called when the low-level connection is lost or closed. + + The argument is an exception object or None (the latter + meaning a regular EOF is received or the connection was + aborted or closed). + """ + if self._session_established: + self._session_established = False + self._loop.call_soon(self._app_protocol.connection_lost, exc) + self._transport = None + self._app_transport = None + + def pause_writing(self): + """Called when the low-level transport's buffer goes over + the high-water mark. + """ + self._app_protocol.pause_writing() + + def resume_writing(self): + """Called when the low-level transport's buffer drains below + the low-water mark. + """ + self._app_protocol.resume_writing() + + def data_received(self, data): + """Called when some SSL data is received. + + The argument is a bytes object. + """ + try: + ssldata, appdata = self._sslpipe.feed_ssldata(data) + except ssl.SSLError as e: + if self._loop.get_debug(): + logger.warning('%r: SSL error %s (reason %s)', + self, e.errno, e.reason) + self._abort() + return + + for chunk in ssldata: + self._transport.write(chunk) + + for chunk in appdata: + if chunk: + self._app_protocol.data_received(chunk) + else: + self._start_shutdown() + break + + def eof_received(self): + """Called when the other end of the low-level stream + is half-closed. + + If this returns a false value (including None), the transport + will close itself. If it returns a true value, closing the + transport is up to the protocol. + """ + try: + if self._loop.get_debug(): + logger.debug("%r received EOF", self) + if not self._in_handshake: + keep_open = self._app_protocol.eof_received() + if keep_open: + logger.warning('returning true from eof_received() ' + 'has no effect when using ssl') + finally: + self._transport.close() + + def _get_extra_info(self, name, default=None): + if name in self._extra: + return self._extra[name] + else: + return self._transport.get_extra_info(name, default) + + def _start_shutdown(self): + if self._in_shutdown: + return + self._in_shutdown = True + self._write_appdata(b'') + + def _write_appdata(self, data): + self._write_backlog.append((data, 0)) + self._write_buffer_size += len(data) + self._process_write_backlog() + + def _start_handshake(self): + if self._loop.get_debug(): + logger.debug("%r starts SSL handshake", self) + self._handshake_start_time = self._loop.time() + else: + self._handshake_start_time = None + self._in_handshake = True + # (b'', 1) is a special value in _process_write_backlog() to do + # the SSL handshake + self._write_backlog.append((b'', 1)) + self._loop.call_soon(self._process_write_backlog) + + def _on_handshake_complete(self, handshake_exc): + self._in_handshake = False + + sslobj = self._sslpipe.ssl_object + peercert = None if handshake_exc else sslobj.getpeercert() + try: + if handshake_exc is not None: + raise handshake_exc + if not hasattr(self._sslcontext, 'check_hostname'): + # Verify hostname if requested, Python 3.4+ uses check_hostname + # and checks the hostname in do_handshake() + if (self._server_hostname + and self._sslcontext.verify_mode != ssl.CERT_NONE): + ssl.match_hostname(peercert, self._server_hostname) + except BaseException as exc: + if self._loop.get_debug(): + if isinstance(exc, ssl.CertificateError): + logger.warning("%r: SSL handshake failed " + "on verifying the certificate", + self, exc_info=True) + else: + logger.warning("%r: SSL handshake failed", + self, exc_info=True) + self._transport.close() + if isinstance(exc, Exception): + if self._waiter is not None: + self._waiter.set_exception(exc) + return + else: + raise + + if self._loop.get_debug(): + dt = self._loop.time() - self._handshake_start_time + logger.debug("%r: SSL handshake took %.1f ms", self, dt * 1e3) + + # Add extra info that becomes available after handshake. + self._extra.update(peercert=peercert, + cipher=sslobj.cipher(), + compression=sslobj.compression(), + ) + self._app_protocol.connection_made(self._app_transport) + if self._waiter is not None: + # wait until protocol.connection_made() has been called + self._waiter._set_result_unless_cancelled(None) + self._session_established = True + # In case transport.write() was already called + self._process_write_backlog() + + def _process_write_backlog(self): + # Try to make progress on the write backlog. + if self._transport is None: + return + + try: + for i in range(len(self._write_backlog)): + data, offset = self._write_backlog[0] + if data: + ssldata, offset = self._sslpipe.feed_appdata(data, offset) + elif offset: + ssldata = self._sslpipe.do_handshake(self._on_handshake_complete) + offset = 1 + else: + ssldata = self._sslpipe.shutdown(self._finalize) + offset = 1 + + for chunk in ssldata: + self._transport.write(chunk) + + if offset < len(data): + self._write_backlog[0] = (data, offset) + # A short write means that a write is blocked on a read + # We need to enable reading if it is paused! + assert self._sslpipe.need_ssldata + if self._transport._paused: + self._transport.resume_reading() + break + + # An entire chunk from the backlog was processed. We can + # delete it and reduce the outstanding buffer size. + del self._write_backlog[0] + self._write_buffer_size -= len(data) + except BaseException as exc: + if self._in_handshake: + self._on_handshake_complete(exc) + else: + self._fatal_error(exc, 'Fatal error on SSL transport') + + def _fatal_error(self, exc, message='Fatal error on transport'): + # Should be called from exception handler only. + if isinstance(exc, (BrokenPipeError, ConnectionResetError)): + if self._loop.get_debug(): + logger.debug("%r: %s", self, message, exc_info=True) + else: + self._loop.call_exception_handler({ + 'message': message, + 'exception': exc, + 'transport': self._transport, + 'protocol': self, + }) + if self._transport: + self._transport._force_close(exc) + + def _finalize(self): + if self._transport is not None: + self._transport.close() + + def _abort(self): + if self._transport is not None: + try: + self._transport.abort() + finally: + self._finalize() diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -434,3 +434,8 @@ sock = mock.Mock(socket.socket) sock.gettimeout.return_value = 0.0 return sock + + +def force_legacy_ssl_support(): + return mock.patch('asyncio.sslproto._is_sslproto_available', + return_value=False) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -650,6 +650,10 @@ *httpd.address) self._test_create_ssl_connection(httpd, create_connection) + def test_legacy_create_ssl_connection(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_ssl_connection() + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_ssl_unix_connection(self): @@ -666,6 +670,10 @@ self._test_create_ssl_connection(httpd, create_connection, check_sockname) + def test_legacy_create_ssl_unix_connection(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_ssl_unix_connection() + def test_create_connection_local_addr(self): with test_utils.run_test_server() as httpd: port = support.find_unused_port() @@ -826,6 +834,10 @@ # stop serving server.close() + def test_legacy_create_server_ssl(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_server_ssl() + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_server_ssl(self): @@ -857,6 +869,10 @@ # stop serving server.close() + def test_legacy_create_unix_server_ssl(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_unix_server_ssl() + @unittest.skipIf(ssl is None, 'No ssl module') def test_create_server_ssl_verify_failed(self): proto = MyProto(loop=self.loop) @@ -881,6 +897,10 @@ self.assertIsNone(proto.transport) server.close() + def test_legacy_create_server_ssl_verify_failed(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_server_ssl_verify_failed() + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_server_ssl_verify_failed(self): @@ -907,6 +927,10 @@ self.assertIsNone(proto.transport) server.close() + def test_legacy_create_unix_server_ssl_verify_failed(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_unix_server_ssl_verify_failed() + @unittest.skipIf(ssl is None, 'No ssl module') def test_create_server_ssl_match_failed(self): proto = MyProto(loop=self.loop) @@ -934,6 +958,10 @@ proto.transport.close() server.close() + def test_legacy_create_server_ssl_match_failed(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_server_ssl_match_failed() + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_server_ssl_verified(self): @@ -958,6 +986,11 @@ proto.transport.close() client.close() server.close() + self.loop.run_until_complete(proto.done) + + def test_legacy_create_unix_server_ssl_verified(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_unix_server_ssl_verified() @unittest.skipIf(ssl is None, 'No ssl module') def test_create_server_ssl_verified(self): @@ -982,6 +1015,11 @@ proto.transport.close() client.close() server.close() + self.loop.run_until_complete(proto.done) + + def test_legacy_create_server_ssl_verified(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_server_ssl_verified() def test_create_server_sock(self): proto = asyncio.Future(loop=self.loop) @@ -1746,20 +1784,20 @@ def create_event_loop(self): return asyncio.ProactorEventLoop() - def test_create_ssl_connection(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") - - def test_create_server_ssl(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") - - def test_create_server_ssl_verify_failed(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") - - def test_create_server_ssl_match_failed(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") - - def test_create_server_ssl_verified(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") + def test_legacy_create_ssl_connection(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") + + def test_legacy_create_server_ssl(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") + + def test_legacy_create_server_ssl_verify_failed(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") + + def test_legacy_create_server_ssl_match_failed(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") + + def test_legacy_create_server_ssl_verified(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") def test_reader_callback(self): raise unittest.SkipTest("IocpEventLoop does not have add_reader()") diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -59,9 +59,13 @@ with test_utils.disable_logger(): transport = self.loop._make_ssl_transport( m, asyncio.Protocol(), m, waiter) - self.assertIsInstance(transport, _SelectorSslTransport) + # Sanity check + class_name = transport.__class__.__name__ + self.assertIn("ssl", class_name.lower()) + self.assertIn("transport", class_name.lower()) @mock.patch('asyncio.selector_events.ssl', None) + @mock.patch('asyncio.sslproto.ssl', None) def test_make_ssl_transport_without_ssl_error(self): m = mock.Mock() self.loop.add_reader = mock.Mock() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 00:21:43 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 23:21:43 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNTYw?= =?utf-8?q?=3A_New_SSL_implementation_based_on_ssl=2EMemoryBIO?= Message-ID: <20150113232142.125888.89020@psf.io> https://hg.python.org/cpython/rev/432b817611f2 changeset: 94128:432b817611f2 branch: 3.4 parent: 94126:0387862a5675 user: Victor Stinner date: Wed Jan 14 00:19:09 2015 +0100 summary: Issue #22560: New SSL implementation based on ssl.MemoryBIO The new SSL implementation is based on the new ssl.MemoryBIO which is only available on Python 3.5. On Python 3.4 and older, the legacy SSL implementation (using SSL_write, SSL_read, etc.) is used. The proactor event loop only supports the new implementation. The new asyncio.sslproto module adds _SSLPipe, SSLProtocol and _SSLProtocolTransport classes. _SSLPipe allows to "wrap" or "unwrap" a socket (switch between cleartext and SSL/TLS). Patch written by Antoine Pitrou. sslproto.py is based on gruvi/ssl.py of the gruvi project written by Geert Jansen. This change adds SSL support to ProactorEventLoop on Python 3.5 and newer! It becomes also possible to implement STARTTTLS: switch a cleartext socket to SSL. files: Lib/asyncio/proactor_events.py | 31 +- Lib/asyncio/selector_events.py | 45 +- Lib/asyncio/sslproto.py | 640 ++++++++++ Lib/asyncio/test_utils.py | 5 + Lib/test/test_asyncio/test_events.py | 66 +- Lib/test/test_asyncio/test_selector_events.py | 6 +- 6 files changed, 751 insertions(+), 42 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -11,6 +11,7 @@ from . import base_events from . import constants from . import futures +from . import sslproto from . import transports from .log import logger @@ -367,6 +368,20 @@ return _ProactorSocketTransport(self, sock, protocol, waiter, extra, server) + def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, + *, server_side=False, server_hostname=None, + extra=None, server=None): + if not sslproto._is_sslproto_available(): + raise NotImplementedError("Proactor event loop requires Python 3.5" + " or newer (ssl.MemoryBIO) to support " + "SSL") + + ssl_protocol = sslproto.SSLProtocol(self, protocol, sslcontext, waiter, + server_side, server_hostname) + _ProactorSocketTransport(self, rawsock, ssl_protocol, + extra=extra, server=server) + return ssl_protocol._app_transport + def _make_duplex_pipe_transport(self, sock, protocol, waiter=None, extra=None): return _ProactorDuplexPipeTransport(self, @@ -455,9 +470,8 @@ def _write_to_self(self): self._csock.send(b'\0') - def _start_serving(self, protocol_factory, sock, ssl=None, server=None): - if ssl: - raise ValueError('IocpEventLoop is incompatible with SSL.') + def _start_serving(self, protocol_factory, sock, + sslcontext=None, server=None): def loop(f=None): try: @@ -467,9 +481,14 @@ logger.debug("%r got a new connection from %r: %r", server, addr, conn) protocol = protocol_factory() - self._make_socket_transport( - conn, protocol, - extra={'peername': addr}, server=server) + if sslcontext is not None: + self._make_ssl_transport( + conn, protocol, sslcontext, server_side=True, + extra={'peername': addr}, server=server) + else: + self._make_socket_transport( + conn, protocol, + extra={'peername': addr}, server=server) if self.is_closed(): return f = self._proactor.accept(sock) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -10,6 +10,7 @@ import errno import functools import socket +import sys try: import ssl except ImportError: # pragma: no cover @@ -21,6 +22,7 @@ from . import futures from . import selectors from . import transports +from . import sslproto from .log import logger @@ -58,6 +60,24 @@ def _make_ssl_transport(self, rawsock, protocol, sslcontext, waiter=None, *, server_side=False, server_hostname=None, extra=None, server=None): + if not sslproto._is_sslproto_available(): + return self._make_legacy_ssl_transport( + rawsock, protocol, sslcontext, waiter, + server_side=server_side, server_hostname=server_hostname, + extra=extra, server=server) + + ssl_protocol = sslproto.SSLProtocol(self, protocol, sslcontext, waiter, + server_side, server_hostname) + _SelectorSocketTransport(self, rawsock, ssl_protocol, + extra=extra, server=server) + return ssl_protocol._app_transport + + def _make_legacy_ssl_transport(self, rawsock, protocol, sslcontext, + waiter, *, + server_side=False, server_hostname=None, + extra=None, server=None): + # Use the legacy API: SSL_write, SSL_read, etc. The legacy API is used + # on Python 3.4 and older, when ssl.MemoryBIO is not available. return _SelectorSslTransport( self, rawsock, protocol, sslcontext, waiter, server_side, server_hostname, extra, server) @@ -508,7 +528,8 @@ def _fatal_error(self, exc, message='Fatal error on transport'): # Should be called from exception handler only. - if isinstance(exc, (BrokenPipeError, ConnectionResetError)): + if isinstance(exc, (BrokenPipeError, + ConnectionResetError, ConnectionAbortedError)): if self._loop.get_debug(): logger.debug("%r: %s", self, message, exc_info=True) else: @@ -683,26 +704,8 @@ if ssl is None: raise RuntimeError('stdlib ssl module not available') - if server_side: - if not sslcontext: - raise ValueError('Server side ssl needs a valid SSLContext') - else: - if not sslcontext: - # Client side may pass ssl=True to use a default - # context; in that case the sslcontext passed is None. - # The default is secure for client connections. - if hasattr(ssl, 'create_default_context'): - # Python 3.4+: use up-to-date strong settings. - sslcontext = ssl.create_default_context() - if not server_hostname: - sslcontext.check_hostname = False - else: - # Fallback for Python 3.3. - sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - sslcontext.options |= ssl.OP_NO_SSLv2 - sslcontext.options |= ssl.OP_NO_SSLv3 - sslcontext.set_default_verify_paths() - sslcontext.verify_mode = ssl.CERT_REQUIRED + if not sslcontext: + sslcontext = sslproto._create_transport_context(server_side, server_hostname) wrap_kwargs = { 'server_side': server_side, diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py new file mode 100644 --- /dev/null +++ b/Lib/asyncio/sslproto.py @@ -0,0 +1,640 @@ +import collections +try: + import ssl +except ImportError: # pragma: no cover + ssl = None + +from . import protocols +from . import transports +from .log import logger + + +def _create_transport_context(server_side, server_hostname): + if server_side: + raise ValueError('Server side SSL needs a valid SSLContext') + + # Client side may pass ssl=True to use a default + # context; in that case the sslcontext passed is None. + # The default is secure for client connections. + if hasattr(ssl, 'create_default_context'): + # Python 3.4+: use up-to-date strong settings. + sslcontext = ssl.create_default_context() + if not server_hostname: + sslcontext.check_hostname = False + else: + # Fallback for Python 3.3. + sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + sslcontext.options |= ssl.OP_NO_SSLv2 + sslcontext.options |= ssl.OP_NO_SSLv3 + sslcontext.set_default_verify_paths() + sslcontext.verify_mode = ssl.CERT_REQUIRED + return sslcontext + + +def _is_sslproto_available(): + return hasattr(ssl, "MemoryBIO") + + +# States of an _SSLPipe. +_UNWRAPPED = "UNWRAPPED" +_DO_HANDSHAKE = "DO_HANDSHAKE" +_WRAPPED = "WRAPPED" +_SHUTDOWN = "SHUTDOWN" + + +class _SSLPipe(object): + """An SSL "Pipe". + + An SSL pipe allows you to communicate with an SSL/TLS protocol instance + through memory buffers. It can be used to implement a security layer for an + existing connection where you don't have access to the connection's file + descriptor, or for some reason you don't want to use it. + + An SSL pipe can be in "wrapped" and "unwrapped" mode. In unwrapped mode, + data is passed through untransformed. In wrapped mode, application level + data is encrypted to SSL record level data and vice versa. The SSL record + level is the lowest level in the SSL protocol suite and is what travels + as-is over the wire. + + An SslPipe initially is in "unwrapped" mode. To start SSL, call + do_handshake(). To shutdown SSL again, call unwrap(). + """ + + max_size = 256 * 1024 # Buffer size passed to read() + + def __init__(self, context, server_side, server_hostname=None): + """ + The *context* argument specifies the ssl.SSLContext to use. + + The *server_side* argument indicates whether this is a server side or + client side transport. + + The optional *server_hostname* argument can be used to specify the + hostname you are connecting to. You may only specify this parameter if + the _ssl module supports Server Name Indication (SNI). + """ + self._context = context + self._server_side = server_side + self._server_hostname = server_hostname + self._state = _UNWRAPPED + self._incoming = ssl.MemoryBIO() + self._outgoing = ssl.MemoryBIO() + self._sslobj = None + self._need_ssldata = False + self._handshake_cb = None + self._shutdown_cb = None + + @property + def context(self): + """The SSL context passed to the constructor.""" + return self._context + + @property + def ssl_object(self): + """The internal ssl.SSLObject instance. + + Return None if the pipe is not wrapped. + """ + return self._sslobj + + @property + def need_ssldata(self): + """Whether more record level data is needed to complete a handshake + that is currently in progress.""" + return self._need_ssldata + + @property + def wrapped(self): + """ + Whether a security layer is currently in effect. + + Return False during handshake. + """ + return self._state == _WRAPPED + + def do_handshake(self, callback=None): + """Start the SSL handshake. + + Return a list of ssldata. A ssldata element is a list of buffers + + The optional *callback* argument can be used to install a callback that + will be called when the handshake is complete. The callback will be + called with None if successful, else an exception instance. + """ + if self._state != _UNWRAPPED: + raise RuntimeError('handshake in progress or completed') + self._sslobj = self._context.wrap_bio( + self._incoming, self._outgoing, + server_side=self._server_side, + server_hostname=self._server_hostname) + self._state = _DO_HANDSHAKE + self._handshake_cb = callback + ssldata, appdata = self.feed_ssldata(b'', only_handshake=True) + assert len(appdata) == 0 + return ssldata + + def shutdown(self, callback=None): + """Start the SSL shutdown sequence. + + Return a list of ssldata. A ssldata element is a list of buffers + + The optional *callback* argument can be used to install a callback that + will be called when the shutdown is complete. The callback will be + called without arguments. + """ + if self._state == _UNWRAPPED: + raise RuntimeError('no security layer present') + if self._state == _SHUTDOWN: + raise RuntimeError('shutdown in progress') + assert self._state in (_WRAPPED, _DO_HANDSHAKE) + self._state = _SHUTDOWN + self._shutdown_cb = callback + ssldata, appdata = self.feed_ssldata(b'') + assert appdata == [] or appdata == [b''] + return ssldata + + def feed_eof(self): + """Send a potentially "ragged" EOF. + + This method will raise an SSL_ERROR_EOF exception if the EOF is + unexpected. + """ + self._incoming.write_eof() + ssldata, appdata = self.feed_ssldata(b'') + assert appdata == [] or appdata == [b''] + + def feed_ssldata(self, data, only_handshake=False): + """Feed SSL record level data into the pipe. + + The data must be a bytes instance. It is OK to send an empty bytes + instance. This can be used to get ssldata for a handshake initiated by + this endpoint. + + Return a (ssldata, appdata) tuple. The ssldata element is a list of + buffers containing SSL data that needs to be sent to the remote SSL. + + The appdata element is a list of buffers containing plaintext data that + needs to be forwarded to the application. The appdata list may contain + an empty buffer indicating an SSL "close_notify" alert. This alert must + be acknowledged by calling shutdown(). + """ + if self._state == _UNWRAPPED: + # If unwrapped, pass plaintext data straight through. + if data: + appdata = [data] + else: + appdata = [] + return ([], appdata) + + self._need_ssldata = False + if data: + self._incoming.write(data) + + ssldata = [] + appdata = [] + try: + if self._state == _DO_HANDSHAKE: + # Call do_handshake() until it doesn't raise anymore. + self._sslobj.do_handshake() + self._state = _WRAPPED + if self._handshake_cb: + self._handshake_cb(None) + if only_handshake: + return (ssldata, appdata) + # Handshake done: execute the wrapped block + + if self._state == _WRAPPED: + # Main state: read data from SSL until close_notify + while True: + chunk = self._sslobj.read(self.max_size) + appdata.append(chunk) + if not chunk: # close_notify + break + + elif self._state == _SHUTDOWN: + # Call shutdown() until it doesn't raise anymore. + self._sslobj.unwrap() + self._sslobj = None + self._state = _UNWRAPPED + if self._shutdown_cb: + self._shutdown_cb() + + elif self._state == _UNWRAPPED: + # Drain possible plaintext data after close_notify. + appdata.append(self._incoming.read()) + except (ssl.SSLError, ssl.CertificateError) as exc: + if getattr(exc, 'errno', None) not in ( + ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE, + ssl.SSL_ERROR_SYSCALL): + if self._state == _DO_HANDSHAKE and self._handshake_cb: + self._handshake_cb(exc) + raise + self._need_ssldata = (exc.errno == ssl.SSL_ERROR_WANT_READ) + + # Check for record level data that needs to be sent back. + # Happens for the initial handshake and renegotiations. + if self._outgoing.pending: + ssldata.append(self._outgoing.read()) + return (ssldata, appdata) + + def feed_appdata(self, data, offset=0): + """Feed plaintext data into the pipe. + + Return an (ssldata, offset) tuple. The ssldata element is a list of + buffers containing record level data that needs to be sent to the + remote SSL instance. The offset is the number of plaintext bytes that + were processed, which may be less than the length of data. + + NOTE: In case of short writes, this call MUST be retried with the SAME + buffer passed into the *data* argument (i.e. the id() must be the + same). This is an OpenSSL requirement. A further particularity is that + a short write will always have offset == 0, because the _ssl module + does not enable partial writes. And even though the offset is zero, + there will still be encrypted data in ssldata. + """ + assert 0 <= offset <= len(data) + if self._state == _UNWRAPPED: + # pass through data in unwrapped mode + if offset < len(data): + ssldata = [data[offset:]] + else: + ssldata = [] + return (ssldata, len(data)) + + ssldata = [] + view = memoryview(data) + while True: + self._need_ssldata = False + try: + if offset < len(view): + offset += self._sslobj.write(view[offset:]) + except ssl.SSLError as exc: + # It is not allowed to call write() after unwrap() until the + # close_notify is acknowledged. We return the condition to the + # caller as a short write. + if exc.reason == 'PROTOCOL_IS_SHUTDOWN': + exc.errno = ssl.SSL_ERROR_WANT_READ + if exc.errno not in (ssl.SSL_ERROR_WANT_READ, + ssl.SSL_ERROR_WANT_WRITE, + ssl.SSL_ERROR_SYSCALL): + raise + self._need_ssldata = (exc.errno == ssl.SSL_ERROR_WANT_READ) + + # See if there's any record level data back for us. + if self._outgoing.pending: + ssldata.append(self._outgoing.read()) + if offset == len(view) or self._need_ssldata: + break + return (ssldata, offset) + + +class _SSLProtocolTransport(transports._FlowControlMixin, + transports.Transport): + + def __init__(self, loop, ssl_protocol, app_protocol): + self._loop = loop + self._ssl_protocol = ssl_protocol + self._app_protocol = app_protocol + + def get_extra_info(self, name, default=None): + """Get optional transport information.""" + return self._ssl_protocol._get_extra_info(name, default) + + def close(self): + """Close the transport. + + Buffered data will be flushed asynchronously. No more data + will be received. After all buffered data is flushed, the + protocol's connection_lost() method will (eventually) called + with None as its argument. + """ + self._ssl_protocol._start_shutdown() + + def pause_reading(self): + """Pause the receiving end. + + No data will be passed to the protocol's data_received() + method until resume_reading() is called. + """ + self._ssl_protocol._transport.pause_reading() + + def resume_reading(self): + """Resume the receiving end. + + Data received will once again be passed to the protocol's + data_received() method. + """ + self._ssl_protocol._transport.resume_reading() + + def set_write_buffer_limits(self, high=None, low=None): + """Set the high- and low-water limits for write flow control. + + These two values control when to call the protocol's + pause_writing() and resume_writing() methods. If specified, + the low-water limit must be less than or equal to the + high-water limit. Neither value can be negative. + + The defaults are implementation-specific. If only the + high-water limit is given, the low-water limit defaults to a + implementation-specific value less than or equal to the + high-water limit. Setting high to zero forces low to zero as + well, and causes pause_writing() to be called whenever the + buffer becomes non-empty. Setting low to zero causes + resume_writing() to be called only once the buffer is empty. + Use of zero for either limit is generally sub-optimal as it + reduces opportunities for doing I/O and computation + concurrently. + """ + self._ssl_protocol._transport.set_write_buffer_limits(high, low) + + def get_write_buffer_size(self): + """Return the current size of the write buffer.""" + return self._ssl_protocol._transport.get_write_buffer_size() + + def write(self, data): + """Write some data bytes to the transport. + + This does not block; it buffers the data and arranges for it + to be sent out asynchronously. + """ + if not isinstance(data, (bytes, bytearray, memoryview)): + raise TypeError("data: expecting a bytes-like instance, got {!r}" + .format(type(data).__name__)) + if not data: + return + self._ssl_protocol._write_appdata(data) + + def can_write_eof(self): + """Return True if this transport supports write_eof(), False if not.""" + return False + + def abort(self): + """Close the transport immediately. + + Buffered data will be lost. No more data will be received. + The protocol's connection_lost() method will (eventually) be + called with None as its argument. + """ + self._ssl_protocol._abort() + + +class SSLProtocol(protocols.Protocol): + """SSL protocol. + + Implementation of SSL on top of a socket using incoming and outgoing + buffers which are ssl.MemoryBIO objects. + """ + + def __init__(self, loop, app_protocol, sslcontext, waiter, + server_side=False, server_hostname=None): + if ssl is None: + raise RuntimeError('stdlib ssl module not available') + + if not sslcontext: + sslcontext = _create_transport_context(server_side, server_hostname) + + self._server_side = server_side + if server_hostname and not server_side: + self._server_hostname = server_hostname + else: + self._server_hostname = None + self._sslcontext = sslcontext + # SSL-specific extra info. More info are set when the handshake + # completes. + self._extra = dict(sslcontext=sslcontext) + + # App data write buffering + self._write_backlog = collections.deque() + self._write_buffer_size = 0 + + self._waiter = waiter + self._closing = False + self._loop = loop + self._app_protocol = app_protocol + self._app_transport = _SSLProtocolTransport(self._loop, + self, self._app_protocol) + self._sslpipe = None + self._session_established = False + self._in_handshake = False + self._in_shutdown = False + + def connection_made(self, transport): + """Called when the low-level connection is made. + + Start the SSL handshake. + """ + self._transport = transport + self._sslpipe = _SSLPipe(self._sslcontext, + self._server_side, + self._server_hostname) + self._start_handshake() + + def connection_lost(self, exc): + """Called when the low-level connection is lost or closed. + + The argument is an exception object or None (the latter + meaning a regular EOF is received or the connection was + aborted or closed). + """ + if self._session_established: + self._session_established = False + self._loop.call_soon(self._app_protocol.connection_lost, exc) + self._transport = None + self._app_transport = None + + def pause_writing(self): + """Called when the low-level transport's buffer goes over + the high-water mark. + """ + self._app_protocol.pause_writing() + + def resume_writing(self): + """Called when the low-level transport's buffer drains below + the low-water mark. + """ + self._app_protocol.resume_writing() + + def data_received(self, data): + """Called when some SSL data is received. + + The argument is a bytes object. + """ + try: + ssldata, appdata = self._sslpipe.feed_ssldata(data) + except ssl.SSLError as e: + if self._loop.get_debug(): + logger.warning('%r: SSL error %s (reason %s)', + self, e.errno, e.reason) + self._abort() + return + + for chunk in ssldata: + self._transport.write(chunk) + + for chunk in appdata: + if chunk: + self._app_protocol.data_received(chunk) + else: + self._start_shutdown() + break + + def eof_received(self): + """Called when the other end of the low-level stream + is half-closed. + + If this returns a false value (including None), the transport + will close itself. If it returns a true value, closing the + transport is up to the protocol. + """ + try: + if self._loop.get_debug(): + logger.debug("%r received EOF", self) + if not self._in_handshake: + keep_open = self._app_protocol.eof_received() + if keep_open: + logger.warning('returning true from eof_received() ' + 'has no effect when using ssl') + finally: + self._transport.close() + + def _get_extra_info(self, name, default=None): + if name in self._extra: + return self._extra[name] + else: + return self._transport.get_extra_info(name, default) + + def _start_shutdown(self): + if self._in_shutdown: + return + self._in_shutdown = True + self._write_appdata(b'') + + def _write_appdata(self, data): + self._write_backlog.append((data, 0)) + self._write_buffer_size += len(data) + self._process_write_backlog() + + def _start_handshake(self): + if self._loop.get_debug(): + logger.debug("%r starts SSL handshake", self) + self._handshake_start_time = self._loop.time() + else: + self._handshake_start_time = None + self._in_handshake = True + # (b'', 1) is a special value in _process_write_backlog() to do + # the SSL handshake + self._write_backlog.append((b'', 1)) + self._loop.call_soon(self._process_write_backlog) + + def _on_handshake_complete(self, handshake_exc): + self._in_handshake = False + + sslobj = self._sslpipe.ssl_object + peercert = None if handshake_exc else sslobj.getpeercert() + try: + if handshake_exc is not None: + raise handshake_exc + if not hasattr(self._sslcontext, 'check_hostname'): + # Verify hostname if requested, Python 3.4+ uses check_hostname + # and checks the hostname in do_handshake() + if (self._server_hostname + and self._sslcontext.verify_mode != ssl.CERT_NONE): + ssl.match_hostname(peercert, self._server_hostname) + except BaseException as exc: + if self._loop.get_debug(): + if isinstance(exc, ssl.CertificateError): + logger.warning("%r: SSL handshake failed " + "on verifying the certificate", + self, exc_info=True) + else: + logger.warning("%r: SSL handshake failed", + self, exc_info=True) + self._transport.close() + if isinstance(exc, Exception): + if self._waiter is not None: + self._waiter.set_exception(exc) + return + else: + raise + + if self._loop.get_debug(): + dt = self._loop.time() - self._handshake_start_time + logger.debug("%r: SSL handshake took %.1f ms", self, dt * 1e3) + + # Add extra info that becomes available after handshake. + self._extra.update(peercert=peercert, + cipher=sslobj.cipher(), + compression=sslobj.compression(), + ) + self._app_protocol.connection_made(self._app_transport) + if self._waiter is not None: + # wait until protocol.connection_made() has been called + self._waiter._set_result_unless_cancelled(None) + self._session_established = True + # In case transport.write() was already called + self._process_write_backlog() + + def _process_write_backlog(self): + # Try to make progress on the write backlog. + if self._transport is None: + return + + try: + for i in range(len(self._write_backlog)): + data, offset = self._write_backlog[0] + if data: + ssldata, offset = self._sslpipe.feed_appdata(data, offset) + elif offset: + ssldata = self._sslpipe.do_handshake(self._on_handshake_complete) + offset = 1 + else: + ssldata = self._sslpipe.shutdown(self._finalize) + offset = 1 + + for chunk in ssldata: + self._transport.write(chunk) + + if offset < len(data): + self._write_backlog[0] = (data, offset) + # A short write means that a write is blocked on a read + # We need to enable reading if it is paused! + assert self._sslpipe.need_ssldata + if self._transport._paused: + self._transport.resume_reading() + break + + # An entire chunk from the backlog was processed. We can + # delete it and reduce the outstanding buffer size. + del self._write_backlog[0] + self._write_buffer_size -= len(data) + except BaseException as exc: + if self._in_handshake: + self._on_handshake_complete(exc) + else: + self._fatal_error(exc, 'Fatal error on SSL transport') + + def _fatal_error(self, exc, message='Fatal error on transport'): + # Should be called from exception handler only. + if isinstance(exc, (BrokenPipeError, ConnectionResetError)): + if self._loop.get_debug(): + logger.debug("%r: %s", self, message, exc_info=True) + else: + self._loop.call_exception_handler({ + 'message': message, + 'exception': exc, + 'transport': self._transport, + 'protocol': self, + }) + if self._transport: + self._transport._force_close(exc) + + def _finalize(self): + if self._transport is not None: + self._transport.close() + + def _abort(self): + if self._transport is not None: + try: + self._transport.abort() + finally: + self._finalize() diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -434,3 +434,8 @@ sock = mock.Mock(socket.socket) sock.gettimeout.return_value = 0.0 return sock + + +def force_legacy_ssl_support(): + return mock.patch('asyncio.sslproto._is_sslproto_available', + return_value=False) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -650,6 +650,10 @@ *httpd.address) self._test_create_ssl_connection(httpd, create_connection) + def test_legacy_create_ssl_connection(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_ssl_connection() + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_ssl_unix_connection(self): @@ -666,6 +670,10 @@ self._test_create_ssl_connection(httpd, create_connection, check_sockname) + def test_legacy_create_ssl_unix_connection(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_ssl_unix_connection() + def test_create_connection_local_addr(self): with test_utils.run_test_server() as httpd: port = support.find_unused_port() @@ -826,6 +834,10 @@ # stop serving server.close() + def test_legacy_create_server_ssl(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_server_ssl() + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_server_ssl(self): @@ -857,6 +869,10 @@ # stop serving server.close() + def test_legacy_create_unix_server_ssl(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_unix_server_ssl() + @unittest.skipIf(ssl is None, 'No ssl module') def test_create_server_ssl_verify_failed(self): proto = MyProto(loop=self.loop) @@ -881,6 +897,10 @@ self.assertIsNone(proto.transport) server.close() + def test_legacy_create_server_ssl_verify_failed(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_server_ssl_verify_failed() + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_server_ssl_verify_failed(self): @@ -907,6 +927,10 @@ self.assertIsNone(proto.transport) server.close() + def test_legacy_create_unix_server_ssl_verify_failed(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_unix_server_ssl_verify_failed() + @unittest.skipIf(ssl is None, 'No ssl module') def test_create_server_ssl_match_failed(self): proto = MyProto(loop=self.loop) @@ -934,6 +958,10 @@ proto.transport.close() server.close() + def test_legacy_create_server_ssl_match_failed(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_server_ssl_match_failed() + @unittest.skipIf(ssl is None, 'No ssl module') @unittest.skipUnless(hasattr(socket, 'AF_UNIX'), 'No UNIX Sockets') def test_create_unix_server_ssl_verified(self): @@ -958,6 +986,11 @@ proto.transport.close() client.close() server.close() + self.loop.run_until_complete(proto.done) + + def test_legacy_create_unix_server_ssl_verified(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_unix_server_ssl_verified() @unittest.skipIf(ssl is None, 'No ssl module') def test_create_server_ssl_verified(self): @@ -982,6 +1015,11 @@ proto.transport.close() client.close() server.close() + self.loop.run_until_complete(proto.done) + + def test_legacy_create_server_ssl_verified(self): + with test_utils.force_legacy_ssl_support(): + self.test_create_server_ssl_verified() def test_create_server_sock(self): proto = asyncio.Future(loop=self.loop) @@ -1746,20 +1784,20 @@ def create_event_loop(self): return asyncio.ProactorEventLoop() - def test_create_ssl_connection(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") - - def test_create_server_ssl(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") - - def test_create_server_ssl_verify_failed(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") - - def test_create_server_ssl_match_failed(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") - - def test_create_server_ssl_verified(self): - raise unittest.SkipTest("IocpEventLoop incompatible with SSL") + def test_legacy_create_ssl_connection(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") + + def test_legacy_create_server_ssl(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") + + def test_legacy_create_server_ssl_verify_failed(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") + + def test_legacy_create_server_ssl_match_failed(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") + + def test_legacy_create_server_ssl_verified(self): + raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") def test_reader_callback(self): raise unittest.SkipTest("IocpEventLoop does not have add_reader()") diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -59,9 +59,13 @@ with test_utils.disable_logger(): transport = self.loop._make_ssl_transport( m, asyncio.Protocol(), m, waiter) - self.assertIsInstance(transport, _SelectorSslTransport) + # Sanity check + class_name = transport.__class__.__name__ + self.assertIn("ssl", class_name.lower()) + self.assertIn("transport", class_name.lower()) @mock.patch('asyncio.selector_events.ssl', None) + @mock.patch('asyncio.sslproto.ssl', None) def test_make_ssl_transport_without_ssl_error(self): m = mock.Mock() self.loop.add_reader = mock.Mock() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 00:30:48 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 23:30:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322560=2C_asyncio_?= =?utf-8?q?doc=3A_ProactorEventLoop_now_supports_SSL!?= Message-ID: <20150113233029.72575.56307@psf.io> https://hg.python.org/cpython/rev/b9fbbe7103e7 changeset: 94130:b9fbbe7103e7 user: Victor Stinner date: Wed Jan 14 00:30:22 2015 +0100 summary: Issue #22560, asyncio doc: ProactorEventLoop now supports SSL! files: Doc/library/asyncio-eventloops.rst | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst --- a/Doc/library/asyncio-eventloops.rst +++ b/Doc/library/asyncio-eventloops.rst @@ -100,8 +100,6 @@ :class:`ProactorEventLoop` specific limits: -- SSL is not supported: :meth:`~BaseEventLoop.create_connection` and - :meth:`~BaseEventLoop.create_server` cannot be used with SSL for example - :meth:`~BaseEventLoop.create_datagram_endpoint` (UDP) is not supported - :meth:`~BaseEventLoop.add_reader` and :meth:`~BaseEventLoop.add_writer` are not supported @@ -112,6 +110,10 @@ `_) and on the Windows configuration. See :ref:`asyncio delayed calls `. +.. versionchanged:: 3.5 + + :class:`ProactorEventLoop` now supports SSL. + Mac OS X ^^^^^^^^ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 00:54:45 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 23:54:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150113235437.22417.55706@psf.io> https://hg.python.org/cpython/rev/9e5d35ee0903 changeset: 94132:9e5d35ee0903 parent: 94130:b9fbbe7103e7 parent: 94131:94a6f9a3580e user: Victor Stinner date: Wed Jan 14 00:54:00 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/streams.py | 47 ++++++++++++++--------------- 1 files changed, 22 insertions(+), 25 deletions(-) diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -313,8 +313,8 @@ else: self._loop = loop self._buffer = bytearray() - self._eof = False # Whether we're done. - self._waiter = None # A future. + self._eof = False # Whether we're done. + self._waiter = None # A future used by _wait_for_data() self._exception = None self._transport = None self._paused = False @@ -331,6 +331,14 @@ if not waiter.cancelled(): waiter.set_exception(exc) + def _wakeup_waiter(self): + """Wakeup read() or readline() function waiting for data or EOF.""" + waiter = self._waiter + if waiter is not None: + self._waiter = None + if not waiter.cancelled(): + waiter.set_result(None) + def set_transport(self, transport): assert self._transport is None, 'Transport already set' self._transport = transport @@ -342,11 +350,7 @@ def feed_eof(self): self._eof = True - waiter = self._waiter - if waiter is not None: - self._waiter = None - if not waiter.cancelled(): - waiter.set_result(True) + self._wakeup_waiter() def at_eof(self): """Return True if the buffer is empty and 'feed_eof' was called.""" @@ -359,12 +363,7 @@ return self._buffer.extend(data) - - waiter = self._waiter - if waiter is not None: - self._waiter = None - if not waiter.cancelled(): - waiter.set_result(False) + self._wakeup_waiter() if (self._transport is not None and not self._paused and @@ -379,7 +378,8 @@ else: self._paused = True - def _create_waiter(self, func_name): + def _wait_for_data(self, func_name): + """Wait until feed_data() or feed_eof() is called.""" # StreamReader uses a future to link the protocol feed_data() method # to a read coroutine. Running two read coroutines at the same time # would have an unexpected behaviour. It would not possible to know @@ -387,7 +387,12 @@ if self._waiter is not None: raise RuntimeError('%s() called while another coroutine is ' 'already waiting for incoming data' % func_name) - return futures.Future(loop=self._loop) + + self._waiter = futures.Future(loop=self._loop) + try: + yield from self._waiter + finally: + self._waiter = None @coroutine def readline(self): @@ -417,11 +422,7 @@ break if not_enough: - self._waiter = self._create_waiter('readline') - try: - yield from self._waiter - finally: - self._waiter = None + yield from self._wait_for_data('readline') self._maybe_resume_transport() return bytes(line) @@ -448,11 +449,7 @@ return b''.join(blocks) else: if not self._buffer and not self._eof: - self._waiter = self._create_waiter('read') - try: - yield from self._waiter - finally: - self._waiter = None + yield from self._wait_for_data('read') if n < 0 or len(self._buffer) <= n: data = bytes(self._buffer) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 00:54:45 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 13 Jan 2015 23:54:45 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTk4?= =?utf-8?q?=3A_Reactor_asyncio=2EStreamReader?= Message-ID: <20150113235437.22405.80279@psf.io> https://hg.python.org/cpython/rev/94a6f9a3580e changeset: 94131:94a6f9a3580e branch: 3.4 parent: 94128:432b817611f2 user: Victor Stinner date: Wed Jan 14 00:53:37 2015 +0100 summary: Issue #23198: Reactor asyncio.StreamReader - Add a new _wakeup_waiter() method - Replace _create_waiter() method with a _wait_for_data() coroutine function - Use the value None instead of True or False to wake up the waiter files: Lib/asyncio/streams.py | 47 ++++++++++++++--------------- 1 files changed, 22 insertions(+), 25 deletions(-) diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -313,8 +313,8 @@ else: self._loop = loop self._buffer = bytearray() - self._eof = False # Whether we're done. - self._waiter = None # A future. + self._eof = False # Whether we're done. + self._waiter = None # A future used by _wait_for_data() self._exception = None self._transport = None self._paused = False @@ -331,6 +331,14 @@ if not waiter.cancelled(): waiter.set_exception(exc) + def _wakeup_waiter(self): + """Wakeup read() or readline() function waiting for data or EOF.""" + waiter = self._waiter + if waiter is not None: + self._waiter = None + if not waiter.cancelled(): + waiter.set_result(None) + def set_transport(self, transport): assert self._transport is None, 'Transport already set' self._transport = transport @@ -342,11 +350,7 @@ def feed_eof(self): self._eof = True - waiter = self._waiter - if waiter is not None: - self._waiter = None - if not waiter.cancelled(): - waiter.set_result(True) + self._wakeup_waiter() def at_eof(self): """Return True if the buffer is empty and 'feed_eof' was called.""" @@ -359,12 +363,7 @@ return self._buffer.extend(data) - - waiter = self._waiter - if waiter is not None: - self._waiter = None - if not waiter.cancelled(): - waiter.set_result(False) + self._wakeup_waiter() if (self._transport is not None and not self._paused and @@ -379,7 +378,8 @@ else: self._paused = True - def _create_waiter(self, func_name): + def _wait_for_data(self, func_name): + """Wait until feed_data() or feed_eof() is called.""" # StreamReader uses a future to link the protocol feed_data() method # to a read coroutine. Running two read coroutines at the same time # would have an unexpected behaviour. It would not possible to know @@ -387,7 +387,12 @@ if self._waiter is not None: raise RuntimeError('%s() called while another coroutine is ' 'already waiting for incoming data' % func_name) - return futures.Future(loop=self._loop) + + self._waiter = futures.Future(loop=self._loop) + try: + yield from self._waiter + finally: + self._waiter = None @coroutine def readline(self): @@ -417,11 +422,7 @@ break if not_enough: - self._waiter = self._create_waiter('readline') - try: - yield from self._waiter - finally: - self._waiter = None + yield from self._wait_for_data('readline') self._maybe_resume_transport() return bytes(line) @@ -448,11 +449,7 @@ return b''.join(blocks) else: if not self._buffer and not self._eof: - self._waiter = self._create_waiter('read') - try: - yield from self._waiter - finally: - self._waiter = None + yield from self._wait_for_data('read') if n < 0 or len(self._buffer) <= n: data = bytes(self._buffer) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 02:15:03 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 01:15:03 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150114011446.125880.57496@psf.io> https://hg.python.org/cpython/rev/c47fe739f9dd changeset: 94134:c47fe739f9dd parent: 94132:9e5d35ee0903 parent: 94133:1eae3b6fbec6 user: Victor Stinner date: Wed Jan 14 02:13:51 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_subprocess.py | 73 ++++++++--- Lib/asyncio/subprocess.py | 16 ++- Lib/test/test_asyncio/test_subprocess.py | 36 +++++ 3 files changed, 100 insertions(+), 25 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -96,32 +96,61 @@ def kill(self): self._proc.kill() + def _kill_wait(self): + """Close pipes, kill the subprocess and read its return status. + + Function called when an exception is raised during the creation + of a subprocess. + """ + if self._loop.get_debug(): + logger.warning('Exception during subprocess creation, ' + 'kill the subprocess %r', + self, + exc_info=True) + + proc = self._proc + if proc.stdout: + proc.stdout.close() + if proc.stderr: + proc.stderr.close() + if proc.stdin: + proc.stdin.close() + try: + proc.kill() + except ProcessLookupError: + pass + proc.wait() + @coroutine def _post_init(self): - proc = self._proc - loop = self._loop - if proc.stdin is not None: - _, pipe = yield from loop.connect_write_pipe( - lambda: WriteSubprocessPipeProto(self, 0), - proc.stdin) - self._pipes[0] = pipe - if proc.stdout is not None: - _, pipe = yield from loop.connect_read_pipe( - lambda: ReadSubprocessPipeProto(self, 1), - proc.stdout) - self._pipes[1] = pipe - if proc.stderr is not None: - _, pipe = yield from loop.connect_read_pipe( - lambda: ReadSubprocessPipeProto(self, 2), - proc.stderr) - self._pipes[2] = pipe + try: + proc = self._proc + loop = self._loop + if proc.stdin is not None: + _, pipe = yield from loop.connect_write_pipe( + lambda: WriteSubprocessPipeProto(self, 0), + proc.stdin) + self._pipes[0] = pipe + if proc.stdout is not None: + _, pipe = yield from loop.connect_read_pipe( + lambda: ReadSubprocessPipeProto(self, 1), + proc.stdout) + self._pipes[1] = pipe + if proc.stderr is not None: + _, pipe = yield from loop.connect_read_pipe( + lambda: ReadSubprocessPipeProto(self, 2), + proc.stderr) + self._pipes[2] = pipe - assert self._pending_calls is not None + assert self._pending_calls is not None - self._loop.call_soon(self._protocol.connection_made, self) - for callback, data in self._pending_calls: - self._loop.call_soon(callback, *data) - self._pending_calls = None + self._loop.call_soon(self._protocol.connection_made, self) + for callback, data in self._pending_calls: + self._loop.call_soon(callback, *data) + self._pending_calls = None + except: + self._kill_wait() + raise def _call(self, cb, *data): if self._pending_calls is not None: diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -60,7 +60,9 @@ protocol=self, reader=None, loop=self._loop) - self.waiter.set_result(None) + + if not self.waiter.cancelled(): + self.waiter.set_result(None) def pipe_data_received(self, fd, data): if fd == 1: @@ -216,7 +218,11 @@ protocol_factory, cmd, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - yield from protocol.waiter + try: + yield from protocol.waiter + except: + transport._kill_wait() + raise return Process(transport, protocol, loop) @coroutine @@ -232,5 +238,9 @@ program, *args, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - yield from protocol.waiter + try: + yield from protocol.waiter + except: + transport._kill_wait() + raise return Process(transport, protocol, loop) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -251,6 +251,42 @@ self.loop.run_until_complete(cancel_wait()) + def test_cancel_make_subprocess_transport_exec(self): + @asyncio.coroutine + def cancel_make_transport(): + coro = asyncio.create_subprocess_exec(*PROGRAM_BLOCKED, + loop=self.loop) + task = self.loop.create_task(coro) + + self.loop.call_soon(task.cancel) + try: + yield from task + except asyncio.CancelledError: + pass + + # ignore the log: + # "Exception during subprocess creation, kill the subprocess" + with test_utils.disable_logger(): + self.loop.run_until_complete(cancel_make_transport()) + + def test_cancel_post_init(self): + @asyncio.coroutine + def cancel_make_transport(): + coro = self.loop.subprocess_exec(asyncio.SubprocessProtocol, + *PROGRAM_BLOCKED) + task = self.loop.create_task(coro) + + self.loop.call_soon(task.cancel) + try: + yield from task + except asyncio.CancelledError: + pass + + # ignore the log: + # "Exception during subprocess creation, kill the subprocess" + with test_utils.disable_logger(): + self.loop.run_until_complete(cancel_make_transport()) + if sys.platform != 'win32': # Unix -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 02:15:03 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 01:15:03 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Python_issue_?= =?utf-8?q?=2323173=3A_sync_with_Tulip?= Message-ID: <20150114011446.125886.76755@psf.io> https://hg.python.org/cpython/rev/1eae3b6fbec6 changeset: 94133:1eae3b6fbec6 branch: 3.4 parent: 94131:94a6f9a3580e user: Victor Stinner date: Wed Jan 14 02:10:33 2015 +0100 summary: Python issue #23173: sync with Tulip * If an exception is raised during the creation of a subprocess, kill the subprocess (close pipes, kill and read the return status). Log an error in such case. * Fix SubprocessStreamProtocol.connection_made() to handle cancelled waiter. Add unit test cancelling subprocess methods. files: Lib/asyncio/base_subprocess.py | 73 ++++++++--- Lib/asyncio/subprocess.py | 16 ++- Lib/test/test_asyncio/test_subprocess.py | 36 +++++ 3 files changed, 100 insertions(+), 25 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -96,32 +96,61 @@ def kill(self): self._proc.kill() + def _kill_wait(self): + """Close pipes, kill the subprocess and read its return status. + + Function called when an exception is raised during the creation + of a subprocess. + """ + if self._loop.get_debug(): + logger.warning('Exception during subprocess creation, ' + 'kill the subprocess %r', + self, + exc_info=True) + + proc = self._proc + if proc.stdout: + proc.stdout.close() + if proc.stderr: + proc.stderr.close() + if proc.stdin: + proc.stdin.close() + try: + proc.kill() + except ProcessLookupError: + pass + proc.wait() + @coroutine def _post_init(self): - proc = self._proc - loop = self._loop - if proc.stdin is not None: - _, pipe = yield from loop.connect_write_pipe( - lambda: WriteSubprocessPipeProto(self, 0), - proc.stdin) - self._pipes[0] = pipe - if proc.stdout is not None: - _, pipe = yield from loop.connect_read_pipe( - lambda: ReadSubprocessPipeProto(self, 1), - proc.stdout) - self._pipes[1] = pipe - if proc.stderr is not None: - _, pipe = yield from loop.connect_read_pipe( - lambda: ReadSubprocessPipeProto(self, 2), - proc.stderr) - self._pipes[2] = pipe + try: + proc = self._proc + loop = self._loop + if proc.stdin is not None: + _, pipe = yield from loop.connect_write_pipe( + lambda: WriteSubprocessPipeProto(self, 0), + proc.stdin) + self._pipes[0] = pipe + if proc.stdout is not None: + _, pipe = yield from loop.connect_read_pipe( + lambda: ReadSubprocessPipeProto(self, 1), + proc.stdout) + self._pipes[1] = pipe + if proc.stderr is not None: + _, pipe = yield from loop.connect_read_pipe( + lambda: ReadSubprocessPipeProto(self, 2), + proc.stderr) + self._pipes[2] = pipe - assert self._pending_calls is not None + assert self._pending_calls is not None - self._loop.call_soon(self._protocol.connection_made, self) - for callback, data in self._pending_calls: - self._loop.call_soon(callback, *data) - self._pending_calls = None + self._loop.call_soon(self._protocol.connection_made, self) + for callback, data in self._pending_calls: + self._loop.call_soon(callback, *data) + self._pending_calls = None + except: + self._kill_wait() + raise def _call(self, cb, *data): if self._pending_calls is not None: diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -60,7 +60,9 @@ protocol=self, reader=None, loop=self._loop) - self.waiter.set_result(None) + + if not self.waiter.cancelled(): + self.waiter.set_result(None) def pipe_data_received(self, fd, data): if fd == 1: @@ -216,7 +218,11 @@ protocol_factory, cmd, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - yield from protocol.waiter + try: + yield from protocol.waiter + except: + transport._kill_wait() + raise return Process(transport, protocol, loop) @coroutine @@ -232,5 +238,9 @@ program, *args, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - yield from protocol.waiter + try: + yield from protocol.waiter + except: + transport._kill_wait() + raise return Process(transport, protocol, loop) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -251,6 +251,42 @@ self.loop.run_until_complete(cancel_wait()) + def test_cancel_make_subprocess_transport_exec(self): + @asyncio.coroutine + def cancel_make_transport(): + coro = asyncio.create_subprocess_exec(*PROGRAM_BLOCKED, + loop=self.loop) + task = self.loop.create_task(coro) + + self.loop.call_soon(task.cancel) + try: + yield from task + except asyncio.CancelledError: + pass + + # ignore the log: + # "Exception during subprocess creation, kill the subprocess" + with test_utils.disable_logger(): + self.loop.run_until_complete(cancel_make_transport()) + + def test_cancel_post_init(self): + @asyncio.coroutine + def cancel_make_transport(): + coro = self.loop.subprocess_exec(asyncio.SubprocessProtocol, + *PROGRAM_BLOCKED) + task = self.loop.create_task(coro) + + self.loop.call_soon(task.cancel) + try: + yield from task + except asyncio.CancelledError: + pass + + # ignore the log: + # "Exception during subprocess creation, kill the subprocess" + with test_utils.disable_logger(): + self.loop.run_until_complete(cancel_make_transport()) + if sys.platform != 'win32': # Unix -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 07:57:51 2015 From: python-checkins at python.org (raymond.hettinger) Date: Wed, 14 Jan 2015 06:57:51 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_No_need_to_rebuild_a_const?= =?utf-8?q?ant_dictionary_on_every_call=2E__Move_convert_mapping?= Message-ID: <20150114065744.72555.53906@psf.io> https://hg.python.org/cpython/rev/f99ed1331952 changeset: 94135:f99ed1331952 user: Raymond Hettinger date: Tue Jan 13 22:57:35 2015 -0800 summary: No need to rebuild a constant dictionary on every call. Move convert mapping to module level. files: Lib/functools.py | 33 +++++++++++++++++---------------- 1 files changed, 17 insertions(+), 16 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -174,28 +174,29 @@ return op_result return not op_result +_convert = { + '__lt__': [('__gt__', _gt_from_lt), + ('__le__', _le_from_lt), + ('__ge__', _ge_from_lt)], + '__le__': [('__ge__', _ge_from_le), + ('__lt__', _lt_from_le), + ('__gt__', _gt_from_le)], + '__gt__': [('__lt__', _lt_from_gt), + ('__ge__', _ge_from_gt), + ('__le__', _le_from_gt)], + '__ge__': [('__le__', _le_from_ge), + ('__gt__', _gt_from_ge), + ('__lt__', _lt_from_ge)] +} + def total_ordering(cls): """Class decorator that fills in missing ordering methods""" - convert = { - '__lt__': [('__gt__', _gt_from_lt), - ('__le__', _le_from_lt), - ('__ge__', _ge_from_lt)], - '__le__': [('__ge__', _ge_from_le), - ('__lt__', _lt_from_le), - ('__gt__', _gt_from_le)], - '__gt__': [('__lt__', _lt_from_gt), - ('__ge__', _ge_from_gt), - ('__le__', _le_from_gt)], - '__ge__': [('__le__', _le_from_ge), - ('__gt__', _gt_from_ge), - ('__lt__', _lt_from_ge)] - } # Find user-defined comparisons (not those inherited from object). - roots = [op for op in convert if getattr(cls, op, None) is not getattr(object, op, None)] + roots = [op for op in _convert if getattr(cls, op, None) is not getattr(object, op, None)] if not roots: raise ValueError('must define at least one ordering operation: < > <= >=') root = max(roots) # prefer __lt__ to __le__ to __gt__ to __ge__ - for opname, opfunc in convert[root]: + for opname, opfunc in _convert[root]: if opname not in roots: opfunc.__name__ = opname setattr(cls, opname, opfunc) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 08:26:46 2015 From: python-checkins at python.org (georg.brandl) Date: Wed, 14 Jan 2015 07:26:46 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2323181=3A_codepoi?= =?utf-8?q?nt_-=3E_code_point?= Message-ID: <20150114072635.125880.66551@psf.io> https://hg.python.org/cpython/rev/c917ba25c007 changeset: 94136:c917ba25c007 user: Georg Brandl date: Wed Jan 14 08:26:30 2015 +0100 summary: Closes #23181: codepoint -> code point files: Doc/c-api/unicode.rst | 2 +- Doc/library/codecs.rst | 12 ++++++------ Doc/library/email.mime.rst | 2 +- Doc/library/functions.rst | 2 +- Doc/library/html.entities.rst | 4 ++-- Doc/tutorial/datastructures.rst | 2 +- Doc/whatsnew/3.3.rst | 18 +++++++++--------- 7 files changed, 21 insertions(+), 21 deletions(-) diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -1141,7 +1141,7 @@ mark (U+FEFF). In the other two modes, no BOM mark is prepended. If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output - as a single codepoint. + as a single code point. Return *NULL* if an exception was raised by the codec. diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -841,7 +841,7 @@ Encodings and Unicode --------------------- -Strings are stored internally as sequences of codepoints in +Strings are stored internally as sequences of code points in range ``0x0``-``0x10FFFF``. (See :pep:`393` for more details about the implementation.) Once a string object is used outside of CPU and memory, endianness @@ -852,23 +852,23 @@ collectivity referred to as :term:`text encodings `. The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps -the codepoints 0-255 to the bytes ``0x0``-``0xff``, which means that a string -object that contains codepoints above ``U+00FF`` can't be encoded with this +the code points 0-255 to the bytes ``0x0``-``0xff``, which means that a string +object that contains code points above ``U+00FF`` can't be encoded with this codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks like the following (although the details of the error message may differ): ``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in position 3: ordinal not in range(256)``. There's another group of encodings (the so called charmap encodings) that choose -a different subset of all Unicode code points and how these codepoints are +a different subset of all Unicode code points and how these code points are mapped to the bytes ``0x0``-``0xff``. To see how this is done simply open e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on Windows). There's a string constant with 256 characters that shows you which character is mapped to which byte value. -All of these encodings can only encode 256 of the 1114112 codepoints +All of these encodings can only encode 256 of the 1114112 code points defined in Unicode. A simple and straightforward way that can store each Unicode -code point, is to store each codepoint as four consecutive bytes. There are two +code point, is to store each code point as four consecutive bytes. There are two possibilities: store the bytes in big endian or in little endian order. These two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their disadvantage is that if e.g. you use ``UTF-32-BE`` on a little endian machine you diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -194,7 +194,7 @@ minor type and defaults to :mimetype:`plain`. *_charset* is the character set of the text and is passed as an argument to the :class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults - to ``us-ascii`` if the string contains only ``ascii`` codepoints, and + to ``us-ascii`` if the string contains only ``ascii`` code points, and ``utf-8`` otherwise. The *_charset* parameter accepts either a string or a :class:`~email.charset.Charset` instance. diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -156,7 +156,7 @@ .. function:: chr(i) - Return the string representing a character whose Unicode codepoint is the + Return the string representing a character whose Unicode code point is the integer *i*. For example, ``chr(97)`` returns the string ``'a'``, while ``chr(931)`` returns the string ``'?'``. This is the inverse of :func:`ord`. diff --git a/Doc/library/html.entities.rst b/Doc/library/html.entities.rst --- a/Doc/library/html.entities.rst +++ b/Doc/library/html.entities.rst @@ -33,12 +33,12 @@ .. data:: name2codepoint - A dictionary that maps HTML entity names to the Unicode codepoints. + A dictionary that maps HTML entity names to the Unicode code points. .. data:: codepoint2name - A dictionary that maps Unicode codepoints to HTML entity names. + A dictionary that maps Unicode code points to HTML entity names. .. rubric:: Footnotes diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -685,7 +685,7 @@ all items of two sequences compare equal, the sequences are considered equal. If one sequence is an initial sub-sequence of the other, the shorter sequence is the smaller (lesser) one. Lexicographical ordering for strings uses the Unicode -codepoint number to order individual characters. Some examples of comparisons +code point number to order individual characters. Some examples of comparisons between sequences of the same type:: (1, 2, 3) < (1, 2, 4) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -228,7 +228,7 @@ Changes introduced by :pep:`393` are the following: -* Python now always supports the full range of Unicode codepoints, including +* Python now always supports the full range of Unicode code points, including non-BMP ones (i.e. from ``U+0000`` to ``U+10FFFF``). The distinction between narrow and wide builds no longer exists and Python now behaves like a wide build, even under Windows. @@ -246,7 +246,7 @@ so ``'\U0010FFFF'[0]`` now returns ``'\U0010FFFF'`` and not ``'\uDBFF'``; * all other functions in the standard library now correctly handle - non-BMP codepoints. + non-BMP code points. * The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF`` in hexadecimal). The :c:func:`PyUnicode_GetMax` function still returns @@ -258,13 +258,13 @@ Performance and resource usage ------------------------------ -The storage of Unicode strings now depends on the highest codepoint in the string: - -* pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per codepoint; - -* BMP strings (``U+0000-U+FFFF``) use 2 bytes per codepoint; - -* non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint. +The storage of Unicode strings now depends on the highest code point in the string: + +* pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per code point; + +* BMP strings (``U+0000-U+FFFF``) use 2 bytes per code point; + +* non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per code point. The net effect is that for most applications, memory usage of string storage should decrease significantly - especially compared to former -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 08:41:19 2015 From: python-checkins at python.org (georg.brandl) Date: Wed, 14 Jan 2015 07:41:19 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMyMzE4?= =?utf-8?q?1=3A_codepoint_-=3E_code_point?= Message-ID: <20150114074118.72571.59437@psf.io> https://hg.python.org/cpython/rev/e280a04625cc changeset: 94137:e280a04625cc branch: 2.7 parent: 94123:6a19e37ce94d user: Georg Brandl date: Wed Jan 14 08:26:30 2015 +0100 summary: Closes #23181: codepoint -> code point files: Doc/c-api/unicode.rst | 4 ++-- Doc/library/codecs.rst | 12 ++++++------ Doc/library/htmllib.rst | 4 ++-- Doc/library/json.rst | 2 +- Doc/tutorial/interpreter.rst | 2 +- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -547,7 +547,7 @@ After completion, *\*byteorder* is set to the current byte order at the end of input data. - In a narrow build codepoints outside the BMP will be decoded as surrogate pairs. + In a narrow build code points outside the BMP will be decoded as surrogate pairs. If *byteorder* is *NULL*, the codec starts in native order mode. @@ -580,7 +580,7 @@ mark (U+FEFF). In the other two modes, no BOM mark is prepended. If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output - as a single codepoint. + as a single code point. Return *NULL* if an exception was raised by the codec. diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -787,7 +787,7 @@ Encodings and Unicode --------------------- -Unicode strings are stored internally as sequences of codepoints (to be precise +Unicode strings are stored internally as sequences of code points (to be precise as :c:type:`Py_UNICODE` arrays). Depending on the way Python is compiled (either via ``--enable-unicode=ucs2`` or ``--enable-unicode=ucs4``, with the former being the default) :c:type:`Py_UNICODE` is either a 16-bit or 32-bit data @@ -796,24 +796,24 @@ unicode object into a sequence of bytes is called encoding and recreating the unicode object from the sequence of bytes is known as decoding. There are many different methods for how this transformation can be done (these methods are -also called encodings). The simplest method is to map the codepoints 0-255 to +also called encodings). The simplest method is to map the code points 0-255 to the bytes ``0x0``-``0xff``. This means that a unicode object that contains -codepoints above ``U+00FF`` can't be encoded with this method (which is called +code points above ``U+00FF`` can't be encoded with this method (which is called ``'latin-1'`` or ``'iso-8859-1'``). :func:`unicode.encode` will raise a :exc:`UnicodeEncodeError` that looks like this: ``UnicodeEncodeError: 'latin-1' codec can't encode character u'\u1234' in position 3: ordinal not in range(256)``. There's another group of encodings (the so called charmap encodings) that choose -a different subset of all unicode code points and how these codepoints are +a different subset of all unicode code points and how these code points are mapped to the bytes ``0x0``-``0xff``. To see how this is done simply open e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on Windows). There's a string constant with 256 characters that shows you which character is mapped to which byte value. -All of these encodings can only encode 256 of the 1114112 codepoints +All of these encodings can only encode 256 of the 1114112 code points defined in unicode. A simple and straightforward way that can store each Unicode -code point, is to store each codepoint as four consecutive bytes. There are two +code point, is to store each code point as four consecutive bytes. There are two possibilities: store the bytes in big endian or in little endian order. These two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their disadvantage is that if e.g. you use ``UTF-32-BE`` on a little endian machine you diff --git a/Doc/library/htmllib.rst b/Doc/library/htmllib.rst --- a/Doc/library/htmllib.rst +++ b/Doc/library/htmllib.rst @@ -185,14 +185,14 @@ .. data:: name2codepoint - A dictionary that maps HTML entity names to the Unicode codepoints. + A dictionary that maps HTML entity names to the Unicode code points. .. versionadded:: 2.3 .. data:: codepoint2name - A dictionary that maps Unicode codepoints to HTML entity names. + A dictionary that maps Unicode code points to HTML entity names. .. versionadded:: 2.3 diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -533,7 +533,7 @@ that don't correspond to valid Unicode characters (e.g. unpaired UTF-16 surrogates), but it does note that they may cause interoperability problems. By default, this module accepts and outputs (when present in the original -:class:`str`) codepoints for such sequences. +:class:`str`) code points for such sequences. Infinite and NaN Number Values diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -140,7 +140,7 @@ For example, to write Unicode literals including the Euro currency symbol, the ISO-8859-15 encoding can be used, with the Euro symbol having the ordinal value 164. This script, when saved in the ISO-8859-15 encoding, will print the value -8364 (the Unicode codepoint corresponding to the Euro symbol) and then exit:: +8364 (the Unicode code point corresponding to the Euro symbol) and then exit:: # -*- coding: iso-8859-15 -*- -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Wed Jan 14 10:06:38 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 14 Jan 2015 10:06:38 +0100 Subject: [Python-checkins] Daily reference leaks (c47fe739f9dd): sum=12 Message-ID: results for c47fe739f9dd on branch "default" -------------------------------------------- test_asyncio leaked [3, 0, 0] memory blocks, sum=3 test_collections leaked [4, 0, 0] references, sum=4 test_collections leaked [2, 0, 0] memory blocks, sum=2 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog43SFzV', '-x'] From python-checkins at python.org Wed Jan 14 16:04:35 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 15:04:35 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322038=2C_configur?= =?utf-8?q?e=3A_HAVE=5FSTD=5FATOMIC_now_also_check_that_=22atomic=5Fint=22?= =?utf-8?q?_and?= Message-ID: <20150114150246.125906.80051@psf.io> https://hg.python.org/cpython/rev/dacc944641b1 changeset: 94138:dacc944641b1 parent: 94136:c917ba25c007 user: Victor Stinner date: Wed Jan 14 16:01:46 2015 +0100 summary: Issue #22038, configure: HAVE_STD_ATOMIC now also check that "atomic_int" and "_Atomic void*" types work. Change needed on FreeBSD 10 where stdatomic.h is available but the compiler fails on "_Atomic void*" with "_Atomic cannot be applied to incomplete type 'void'". files: configure | 3 ++- configure.ac | 6 ++++-- pyconfig.h.in | 2 +- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/configure b/configure --- a/configure +++ b/configure @@ -15711,7 +15711,8 @@ #include - _Atomic int value = ATOMIC_VAR_INIT(1); + atomic_int value = ATOMIC_VAR_INIT(1); + _Atomic void *py_atomic_address = (void*) &value; int main() { int loaded_value = atomic_load(&value); return 0; diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -4890,7 +4890,8 @@ [ AC_LANG_SOURCE([[ #include - _Atomic int value = ATOMIC_VAR_INIT(1); + atomic_int value = ATOMIC_VAR_INIT(1); + _Atomic void *py_atomic_address = (void*) &value; int main() { int loaded_value = atomic_load(&value); return 0; @@ -4901,7 +4902,8 @@ AC_MSG_RESULT($have_stdatomic_h) if test "$have_stdatomic_h" = yes; then - AC_DEFINE(HAVE_STD_ATOMIC, 1, [Has stdatomic.h]) + AC_DEFINE(HAVE_STD_ATOMIC, 1, + [Has stdatomic.h, atomic_int and _Atomic void* types work]) fi # Check for GCC >= 4.7 __atomic builtins diff --git a/pyconfig.h.in b/pyconfig.h.in --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -880,7 +880,7 @@ /* Define to 1 if you have the header file. */ #undef HAVE_STDLIB_H -/* Has stdatomic.h */ +/* Has stdatomic.h, atomic_int and _Atomic void* types work */ #undef HAVE_STD_ATOMIC /* Define to 1 if you have the `strdup' function. */ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 17:03:00 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 16:03:00 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150114160121.8751.38471@psf.io> https://hg.python.org/cpython/rev/697f727fa91f changeset: 94140:697f727fa91f parent: 94138:dacc944641b1 parent: 94139:c9ad45b15919 user: Victor Stinner date: Wed Jan 14 17:00:29 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/selector_events.py | 2 +- Lib/asyncio/sslproto.py | 5 +- Lib/test/test_asyncio/test_selector_events.py | 20 +++- Lib/test/test_asyncio/test_sslproto.py | 45 ++++++++++ 4 files changed, 65 insertions(+), 7 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -750,7 +750,7 @@ self._loop.remove_reader(self._sock_fd) self._loop.remove_writer(self._sock_fd) self._sock.close() - if self._waiter is not None: + if self._waiter is not None and not self._waiter.cancelled(): self._waiter.set_exception(exc) if isinstance(exc, Exception): return diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -530,10 +530,11 @@ self._in_handshake = False sslobj = self._sslpipe.ssl_object - peercert = None if handshake_exc else sslobj.getpeercert() try: if handshake_exc is not None: raise handshake_exc + + peercert = sslobj.getpeercert() if not hasattr(self._sslcontext, 'check_hostname'): # Verify hostname if requested, Python 3.4+ uses check_hostname # and checks the hostname in do_handshake() @@ -551,7 +552,7 @@ self, exc_info=True) self._transport.close() if isinstance(exc, Exception): - if self._waiter is not None: + if self._waiter is not None and not self._waiter.cancelled(): self._waiter.set_exception(exc) return else: diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1148,16 +1148,28 @@ self.assertTrue(self.sslsock.close.called) def test_on_handshake_base_exc(self): + waiter = asyncio.Future(loop=self.loop) transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext) - transport._waiter = asyncio.Future(loop=self.loop) + self.loop, self.sock, self.protocol, self.sslcontext, waiter) exc = BaseException() self.sslsock.do_handshake.side_effect = exc with test_utils.disable_logger(): self.assertRaises(BaseException, transport._on_handshake, 0) self.assertTrue(self.sslsock.close.called) - self.assertTrue(transport._waiter.done()) - self.assertIs(exc, transport._waiter.exception()) + self.assertTrue(waiter.done()) + self.assertIs(exc, waiter.exception()) + + def test_cancel_handshake(self): + # Python issue #23197: cancelling an handshake must not raise an + # exception or log an error, even if the handshake failed + waiter = asyncio.Future(loop=self.loop) + transport = _SelectorSslTransport( + self.loop, self.sock, self.protocol, self.sslcontext, waiter) + waiter.cancel() + exc = ValueError() + self.sslsock.do_handshake.side_effect = exc + with test_utils.disable_logger(): + transport._on_handshake(0) def test_pause_resume_reading(self): tr = self._make_one() diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -0,0 +1,45 @@ +"""Tests for asyncio/sslproto.py.""" + +import unittest +from unittest import mock + +import asyncio +from asyncio import sslproto +from asyncio import test_utils + + +class SslProtoHandshakeTests(test_utils.TestCase): + + def setUp(self): + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + + def test_cancel_handshake(self): + # Python issue #23197: cancelling an handshake must not raise an + # exception or log an error, even if the handshake failed + sslcontext = test_utils.dummy_ssl_context() + app_proto = asyncio.Protocol() + waiter = asyncio.Future(loop=self.loop) + ssl_proto = sslproto.SSLProtocol(self.loop, app_proto, sslcontext, + waiter) + handshake_fut = asyncio.Future(loop=self.loop) + + def do_handshake(callback): + exc = Exception() + callback(exc) + handshake_fut.set_result(None) + return [] + + waiter.cancel() + transport = mock.Mock() + sslpipe = mock.Mock() + sslpipe.do_handshake.side_effect = do_handshake + with mock.patch('asyncio.sslproto._SSLPipe', return_value=sslpipe): + ssl_proto.connection_made(transport) + + with test_utils.disable_logger(): + self.loop.run_until_complete(handshake_fut) + + +if __name__ == '__main__': + unittest.main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 17:03:00 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 16:03:00 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTk3?= =?utf-8?q?=2C_asyncio=3A_On_SSL_handshake_failure=2C_check_if_the_waiter_?= =?utf-8?q?is?= Message-ID: <20150114160120.125886.1047@psf.io> https://hg.python.org/cpython/rev/c9ad45b15919 changeset: 94139:c9ad45b15919 branch: 3.4 parent: 94133:1eae3b6fbec6 user: Victor Stinner date: Wed Jan 14 16:56:20 2015 +0100 summary: Issue #23197, asyncio: On SSL handshake failure, check if the waiter is cancelled before setting its exception. * Add unit tests for this case. * Cleanup also sslproto.py files: Lib/asyncio/selector_events.py | 2 +- Lib/asyncio/sslproto.py | 5 +- Lib/test/test_asyncio/test_selector_events.py | 20 +++- Lib/test/test_asyncio/test_sslproto.py | 45 ++++++++++ 4 files changed, 65 insertions(+), 7 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -750,7 +750,7 @@ self._loop.remove_reader(self._sock_fd) self._loop.remove_writer(self._sock_fd) self._sock.close() - if self._waiter is not None: + if self._waiter is not None and not self._waiter.cancelled(): self._waiter.set_exception(exc) if isinstance(exc, Exception): return diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -530,10 +530,11 @@ self._in_handshake = False sslobj = self._sslpipe.ssl_object - peercert = None if handshake_exc else sslobj.getpeercert() try: if handshake_exc is not None: raise handshake_exc + + peercert = sslobj.getpeercert() if not hasattr(self._sslcontext, 'check_hostname'): # Verify hostname if requested, Python 3.4+ uses check_hostname # and checks the hostname in do_handshake() @@ -551,7 +552,7 @@ self, exc_info=True) self._transport.close() if isinstance(exc, Exception): - if self._waiter is not None: + if self._waiter is not None and not self._waiter.cancelled(): self._waiter.set_exception(exc) return else: diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1148,16 +1148,28 @@ self.assertTrue(self.sslsock.close.called) def test_on_handshake_base_exc(self): + waiter = asyncio.Future(loop=self.loop) transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext) - transport._waiter = asyncio.Future(loop=self.loop) + self.loop, self.sock, self.protocol, self.sslcontext, waiter) exc = BaseException() self.sslsock.do_handshake.side_effect = exc with test_utils.disable_logger(): self.assertRaises(BaseException, transport._on_handshake, 0) self.assertTrue(self.sslsock.close.called) - self.assertTrue(transport._waiter.done()) - self.assertIs(exc, transport._waiter.exception()) + self.assertTrue(waiter.done()) + self.assertIs(exc, waiter.exception()) + + def test_cancel_handshake(self): + # Python issue #23197: cancelling an handshake must not raise an + # exception or log an error, even if the handshake failed + waiter = asyncio.Future(loop=self.loop) + transport = _SelectorSslTransport( + self.loop, self.sock, self.protocol, self.sslcontext, waiter) + waiter.cancel() + exc = ValueError() + self.sslsock.do_handshake.side_effect = exc + with test_utils.disable_logger(): + transport._on_handshake(0) def test_pause_resume_reading(self): tr = self._make_one() diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -0,0 +1,45 @@ +"""Tests for asyncio/sslproto.py.""" + +import unittest +from unittest import mock + +import asyncio +from asyncio import sslproto +from asyncio import test_utils + + +class SslProtoHandshakeTests(test_utils.TestCase): + + def setUp(self): + self.loop = asyncio.new_event_loop() + self.set_event_loop(self.loop) + + def test_cancel_handshake(self): + # Python issue #23197: cancelling an handshake must not raise an + # exception or log an error, even if the handshake failed + sslcontext = test_utils.dummy_ssl_context() + app_proto = asyncio.Protocol() + waiter = asyncio.Future(loop=self.loop) + ssl_proto = sslproto.SSLProtocol(self.loop, app_proto, sslcontext, + waiter) + handshake_fut = asyncio.Future(loop=self.loop) + + def do_handshake(callback): + exc = Exception() + callback(exc) + handshake_fut.set_result(None) + return [] + + waiter.cancel() + transport = mock.Mock() + sslpipe = mock.Mock() + sslpipe.do_handshake.side_effect = do_handshake + with mock.patch('asyncio.sslproto._SSLPipe', return_value=sslpipe): + ssl_proto.connection_made(transport) + + with test_utils.disable_logger(): + self.loop.run_until_complete(handshake_fut) + + +if __name__ == '__main__': + unittest.main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 17:11:06 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 16:11:06 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2323234=3A_Refacto?= =?utf-8?q?r_subprocess?= Message-ID: <20150114160850.8739.26565@psf.io> https://hg.python.org/cpython/rev/0c5ae257966f changeset: 94141:0c5ae257966f user: Victor Stinner date: Wed Jan 14 17:07:59 2015 +0100 summary: Closes #23234: Refactor subprocess Use new OSError exceptions, factorize stdin.write() code. files: Lib/subprocess.py | 58 ++++++++++++++-------------------- 1 files changed, 24 insertions(+), 34 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -920,6 +920,22 @@ self._devnull = os.open(os.devnull, os.O_RDWR) return self._devnull + def _stdin_write(self, input): + if input: + try: + self.stdin.write(input) + except BrokenPipeError: + # communicate() must ignore broken pipe error + pass + except OSError as e: + if e.errno == errno.EINVAL and self.poll() is not None: + # Issue #19612: On Windows, stdin.write() fails with EINVAL + # if the process already exited before the write + pass + else: + raise + self.stdin.close() + def communicate(self, input=None, timeout=None): """Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for @@ -945,13 +961,7 @@ stdout = None stderr = None if self.stdin: - if input: - try: - self.stdin.write(input) - except OSError as e: - if e.errno != errno.EPIPE and e.errno != errno.EINVAL: - raise - self.stdin.close() + self._stdin_write(input) elif self.stdout: stdout = _eintr_retry_call(self.stdout.read) self.stdout.close() @@ -1200,21 +1210,7 @@ self.stderr_thread.start() if self.stdin: - if input is not None: - try: - self.stdin.write(input) - except OSError as e: - if e.errno == errno.EPIPE: - # communicate() should ignore pipe full error - pass - elif (e.errno == errno.EINVAL - and self.poll() is not None): - # Issue #19612: stdin.write() fails with EINVAL - # if the process already exited before the write - pass - else: - raise - self.stdin.close() + self._stdin_write(input) # Wait for the reader threads, or time out. If we time out, the # threads remain reading and the fds left open in case the user @@ -1425,9 +1421,8 @@ if errpipe_data: try: _eintr_retry_call(os.waitpid, self.pid, 0) - except OSError as e: - if e.errno != errno.ECHILD: - raise + except ChildProcessError: + pass try: exception_name, hex_errno, err_msg = ( errpipe_data.split(b':', 2)) @@ -1511,9 +1506,7 @@ """All callers to this function MUST hold self._waitpid_lock.""" try: (pid, sts) = _eintr_retry_call(os.waitpid, self.pid, wait_flags) - except OSError as e: - if e.errno != errno.ECHILD: - raise + except ChildProcessError: # This happens if SIGCLD is set to be ignored or waiting # for child processes has otherwise been disabled for our # process. This child is dead, we can't get the status. @@ -1625,12 +1618,9 @@ self._input_offset + _PIPE_BUF] try: self._input_offset += os.write(key.fd, chunk) - except OSError as e: - if e.errno == errno.EPIPE: - selector.unregister(key.fileobj) - key.fileobj.close() - else: - raise + except BrokenPipeError: + selector.unregister(key.fileobj) + key.fileobj.close() else: if self._input_offset >= len(self._input): selector.unregister(key.fileobj) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 17:42:54 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 16:42:54 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTk3?= =?utf-8?q?=3A_On_SSL_handshake_failure_on_matching_hostname=2C_check_if_t?= =?utf-8?q?he?= Message-ID: <20150114161446.11575.3775@psf.io> https://hg.python.org/cpython/rev/42f4dfc6c6a9 changeset: 94142:42f4dfc6c6a9 branch: 3.4 parent: 94139:c9ad45b15919 user: Victor Stinner date: Wed Jan 14 17:13:28 2015 +0100 summary: Issue #23197: On SSL handshake failure on matching hostname, check if the waiter is cancelled before setting its exception. files: Lib/asyncio/selector_events.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -774,7 +774,8 @@ "on matching the hostname", self, exc_info=True) self._sock.close() - if self._waiter is not None: + if (self._waiter is not None + and not self._waiter.cancelled()): self._waiter.set_exception(exc) return -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 17:42:54 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 16:42:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150114161446.72569.93397@psf.io> https://hg.python.org/cpython/rev/c00be209bccf changeset: 94143:c00be209bccf parent: 94141:0c5ae257966f parent: 94142:42f4dfc6c6a9 user: Victor Stinner date: Wed Jan 14 17:13:43 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/selector_events.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -774,7 +774,8 @@ "on matching the hostname", self, exc_info=True) self._sock.close() - if self._waiter is not None: + if (self._waiter is not None + and not self._waiter.cancelled()): self._waiter.set_exception(exc) return -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 14 22:06:04 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 14 Jan 2015 21:06:04 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_add_a_rss_target?= Message-ID: <20150114210545.125890.92216@psf.io> https://hg.python.org/peps/rev/04d75fef530e changeset: 5671:04d75fef530e user: Benjamin Peterson date: Wed Jan 14 16:05:43 2015 -0500 summary: add a rss target files: Makefile | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Makefile b/Makefile --- a/Makefile +++ b/Makefile @@ -21,6 +21,9 @@ pep-0000.txt: $(wildcard pep-????.txt) $(wildcard pep0/*.py) $(PYTHON) genpepindex.py . +rss: + $(PYTHON) pep2rss.py . + install: echo "Installing is not necessary anymore. It will be done in post-commit." -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Thu Jan 15 00:06:37 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 23:06:37 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogc3lu?= =?utf-8?q?c_with_Tulip?= Message-ID: <20150114230630.22407.8324@psf.io> https://hg.python.org/cpython/rev/463bbd862887 changeset: 94144:463bbd862887 branch: 3.4 parent: 94142:42f4dfc6c6a9 user: Victor Stinner date: Thu Jan 15 00:04:21 2015 +0100 summary: asyncio: sync with Tulip * PipeHandle now uses None instead of -1 for a closed handle * Sort imports in windows_utils. * Fix test_events on Python older than 3.5. Skip SSL tests on the ProactorEventLoop if ssl.MemoryIO is missing * Fix BaseEventLoop._create_connection_transport(). Close the transport if the creation of the transport (if the waiter) gets an exception. * _ProactorBasePipeTransport now sets _sock to None when the transport is closed. * Fix BaseSubprocessTransport.close(). Ignore pipes for which the protocol is not set yet (still equal to None). * TestLoop.close() now calls the close() method of the parent class (BaseEventLoop). * Cleanup BaseSelectorEventLoop: create the protocol on a separated line for readability and ease debugging. * Fix BaseSubprocessTransport._kill_wait(). Set the _returncode attribute, so close() doesn't try to terminate the process. * Tests: explicitly close event loops and transports * UNIX pipe transports: add closed/closing in repr(). Add "closed" or "closing" state in the __repr__() method of _UnixReadPipeTransport and _UnixWritePipeTransport classes. files: Lib/asyncio/base_events.py | 7 ++- Lib/asyncio/base_subprocess.py | 4 +- Lib/asyncio/proactor_events.py | 1 + Lib/asyncio/selector_events.py | 5 +- Lib/asyncio/test_utils.py | 1 + Lib/asyncio/unix_events.py | 14 +++++- Lib/asyncio/windows_utils.py | 10 ++-- Lib/test/test_asyncio/test_base_events.py | 1 + Lib/test/test_asyncio/test_events.py | 22 ++++++++++ Lib/test/test_asyncio/test_futures.py | 1 + Lib/test/test_asyncio/test_selector_events.py | 1 + Lib/test/test_asyncio/test_unix_events.py | 1 + 12 files changed, 57 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -634,7 +634,12 @@ else: transport = self._make_socket_transport(sock, protocol, waiter) - yield from waiter + try: + yield from waiter + except Exception as exc: + transport.close() + raise + return transport, protocol @coroutine diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -71,6 +71,8 @@ def close(self): for proto in self._pipes.values(): + if proto is None: + continue proto.pipe.close() if self._returncode is None: self.terminate() @@ -119,7 +121,7 @@ proc.kill() except ProcessLookupError: pass - proc.wait() + self._returncode = proc.wait() @coroutine def _post_init(self): diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -111,6 +111,7 @@ if hasattr(self._sock, 'shutdown'): self._sock.shutdown(socket.SHUT_RDWR) self._sock.close() + self._sock = None server = self._server if server is not None: server._detach() diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -182,13 +182,14 @@ else: raise # The event loop will catch, log and ignore it. else: + protocol = protocol_factory() if sslcontext: self._make_ssl_transport( - conn, protocol_factory(), sslcontext, + conn, protocol, sslcontext, server_side=True, extra={'peername': addr}, server=server) else: self._make_socket_transport( - conn, protocol_factory(), extra={'peername': addr}, + conn, protocol , extra={'peername': addr}, server=server) # It's now up to the protocol to handle the connection. diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -307,6 +307,7 @@ self._time += advance def close(self): + super().close() if self._check_on_close: try: self._gen.send(0) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -301,7 +301,12 @@ self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): - info = [self.__class__.__name__, 'fd=%s' % self._fileno] + info = [self.__class__.__name__] + if self._pipe is None: + info.append('closed') + elif self._closing: + info.append('closing') + info.append('fd=%s' % self._fileno) if self._pipe is not None: polling = selector_events._test_selector_event( self._loop._selector, @@ -404,7 +409,12 @@ self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): - info = [self.__class__.__name__, 'fd=%s' % self._fileno] + info = [self.__class__.__name__] + if self._pipe is None: + info.append('closed') + elif self._closing: + info.append('closing') + info.append('fd=%s' % self._fileno) if self._pipe is not None: polling = selector_events._test_selector_event( self._loop._selector, diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -7,13 +7,13 @@ if sys.platform != 'win32': # pragma: no cover raise ImportError('win32 only') -import socket +import _winapi import itertools import msvcrt import os +import socket import subprocess import tempfile -import _winapi __all__ = ['socketpair', 'pipe', 'Popen', 'PIPE', 'PipeHandle'] @@ -136,7 +136,7 @@ self._handle = handle def __repr__(self): - if self._handle != -1: + if self._handle is not None: handle = 'handle=%r' % self._handle else: handle = 'closed' @@ -150,9 +150,9 @@ return self._handle def close(self, *, CloseHandle=_winapi.CloseHandle): - if self._handle != -1: + if self._handle is not None: CloseHandle(self._handle) - self._handle = -1 + self._handle = None __del__ = close diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -409,6 +409,7 @@ def test_run_until_complete_loop(self): task = asyncio.Future(loop=self.loop) other_loop = self.new_test_loop() + self.addCleanup(other_loop.close) self.assertRaises(ValueError, other_loop.run_until_complete, task) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -25,6 +25,7 @@ import asyncio from asyncio import proactor_events from asyncio import selector_events +from asyncio import sslproto from asyncio import test_utils try: from test import support @@ -1585,6 +1586,7 @@ self.assertTrue(all(f.done() for f in proto.disconnects.values())) self.assertEqual(proto.data[1].rstrip(b'\r\n'), b'Python') self.assertEqual(proto.data[2], b'') + transp.close() def test_subprocess_exitcode(self): connect = self.loop.subprocess_shell( @@ -1594,6 +1596,7 @@ self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.completed) self.assertEqual(7, proto.returncode) + transp.close() def test_subprocess_close_after_finish(self): connect = self.loop.subprocess_shell( @@ -1621,6 +1624,7 @@ transp.kill() self.loop.run_until_complete(proto.completed) self.check_killed(proto.returncode) + transp.close() def test_subprocess_terminate(self): prog = os.path.join(os.path.dirname(__file__), 'echo.py') @@ -1635,6 +1639,7 @@ transp.terminate() self.loop.run_until_complete(proto.completed) self.check_terminated(proto.returncode) + transp.close() @unittest.skipIf(sys.platform == 'win32', "Don't have SIGHUP") def test_subprocess_send_signal(self): @@ -1650,6 +1655,7 @@ transp.send_signal(signal.SIGHUP) self.loop.run_until_complete(proto.completed) self.assertEqual(-signal.SIGHUP, proto.returncode) + transp.close() def test_subprocess_stderr(self): prog = os.path.join(os.path.dirname(__file__), 'echo2.py') @@ -1784,6 +1790,22 @@ def create_event_loop(self): return asyncio.ProactorEventLoop() + if not sslproto._is_sslproto_available(): + def test_create_ssl_connection(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + + def test_create_server_ssl(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + + def test_create_server_ssl_verify_failed(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + + def test_create_server_ssl_match_failed(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + + def test_create_server_ssl_verified(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + def test_legacy_create_ssl_connection(self): raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py --- a/Lib/test/test_asyncio/test_futures.py +++ b/Lib/test/test_asyncio/test_futures.py @@ -29,6 +29,7 @@ def setUp(self): self.loop = self.new_test_loop() + self.addCleanup(self.loop.close) def test_initial_state(self): f = asyncio.Future(loop=self.loop) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1744,6 +1744,7 @@ test_utils.MockPattern( 'Fatal error on transport\nprotocol:.*\ntransport:.*'), exc_info=(ConnectionRefusedError, MOCK_ANY, MOCK_ANY)) + transport.close() if __name__ == '__main__': diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -598,6 +598,7 @@ # This is a bit overspecified. :-( m_log.warning.assert_called_with( 'pipe closed by peer or os.write(pipe, data) raised exception.') + tr.close() @mock.patch('os.write') def test_write_close(self, m_write): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 00:06:37 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 14 Jan 2015 23:06:37 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150114230630.11561.72451@psf.io> https://hg.python.org/cpython/rev/61a045ac0006 changeset: 94145:61a045ac0006 parent: 94143:c00be209bccf parent: 94144:463bbd862887 user: Victor Stinner date: Thu Jan 15 00:05:18 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 7 ++- Lib/asyncio/base_subprocess.py | 4 +- Lib/asyncio/proactor_events.py | 1 + Lib/asyncio/selector_events.py | 5 +- Lib/asyncio/test_utils.py | 1 + Lib/asyncio/unix_events.py | 14 +++++- Lib/asyncio/windows_utils.py | 10 ++-- Lib/test/test_asyncio/test_base_events.py | 1 + Lib/test/test_asyncio/test_events.py | 22 ++++++++++ Lib/test/test_asyncio/test_futures.py | 1 + Lib/test/test_asyncio/test_selector_events.py | 1 + Lib/test/test_asyncio/test_unix_events.py | 1 + 12 files changed, 57 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -634,7 +634,12 @@ else: transport = self._make_socket_transport(sock, protocol, waiter) - yield from waiter + try: + yield from waiter + except Exception as exc: + transport.close() + raise + return transport, protocol @coroutine diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -71,6 +71,8 @@ def close(self): for proto in self._pipes.values(): + if proto is None: + continue proto.pipe.close() if self._returncode is None: self.terminate() @@ -119,7 +121,7 @@ proc.kill() except ProcessLookupError: pass - proc.wait() + self._returncode = proc.wait() @coroutine def _post_init(self): diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -111,6 +111,7 @@ if hasattr(self._sock, 'shutdown'): self._sock.shutdown(socket.SHUT_RDWR) self._sock.close() + self._sock = None server = self._server if server is not None: server._detach() diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -182,13 +182,14 @@ else: raise # The event loop will catch, log and ignore it. else: + protocol = protocol_factory() if sslcontext: self._make_ssl_transport( - conn, protocol_factory(), sslcontext, + conn, protocol, sslcontext, server_side=True, extra={'peername': addr}, server=server) else: self._make_socket_transport( - conn, protocol_factory(), extra={'peername': addr}, + conn, protocol , extra={'peername': addr}, server=server) # It's now up to the protocol to handle the connection. diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -307,6 +307,7 @@ self._time += advance def close(self): + super().close() if self._check_on_close: try: self._gen.send(0) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -301,7 +301,12 @@ self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): - info = [self.__class__.__name__, 'fd=%s' % self._fileno] + info = [self.__class__.__name__] + if self._pipe is None: + info.append('closed') + elif self._closing: + info.append('closing') + info.append('fd=%s' % self._fileno) if self._pipe is not None: polling = selector_events._test_selector_event( self._loop._selector, @@ -404,7 +409,12 @@ self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): - info = [self.__class__.__name__, 'fd=%s' % self._fileno] + info = [self.__class__.__name__] + if self._pipe is None: + info.append('closed') + elif self._closing: + info.append('closing') + info.append('fd=%s' % self._fileno) if self._pipe is not None: polling = selector_events._test_selector_event( self._loop._selector, diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -7,13 +7,13 @@ if sys.platform != 'win32': # pragma: no cover raise ImportError('win32 only') -import socket +import _winapi import itertools import msvcrt import os +import socket import subprocess import tempfile -import _winapi __all__ = ['socketpair', 'pipe', 'Popen', 'PIPE', 'PipeHandle'] @@ -136,7 +136,7 @@ self._handle = handle def __repr__(self): - if self._handle != -1: + if self._handle is not None: handle = 'handle=%r' % self._handle else: handle = 'closed' @@ -150,9 +150,9 @@ return self._handle def close(self, *, CloseHandle=_winapi.CloseHandle): - if self._handle != -1: + if self._handle is not None: CloseHandle(self._handle) - self._handle = -1 + self._handle = None __del__ = close diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -409,6 +409,7 @@ def test_run_until_complete_loop(self): task = asyncio.Future(loop=self.loop) other_loop = self.new_test_loop() + self.addCleanup(other_loop.close) self.assertRaises(ValueError, other_loop.run_until_complete, task) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -25,6 +25,7 @@ import asyncio from asyncio import proactor_events from asyncio import selector_events +from asyncio import sslproto from asyncio import test_utils try: from test import support @@ -1585,6 +1586,7 @@ self.assertTrue(all(f.done() for f in proto.disconnects.values())) self.assertEqual(proto.data[1].rstrip(b'\r\n'), b'Python') self.assertEqual(proto.data[2], b'') + transp.close() def test_subprocess_exitcode(self): connect = self.loop.subprocess_shell( @@ -1594,6 +1596,7 @@ self.assertIsInstance(proto, MySubprocessProtocol) self.loop.run_until_complete(proto.completed) self.assertEqual(7, proto.returncode) + transp.close() def test_subprocess_close_after_finish(self): connect = self.loop.subprocess_shell( @@ -1621,6 +1624,7 @@ transp.kill() self.loop.run_until_complete(proto.completed) self.check_killed(proto.returncode) + transp.close() def test_subprocess_terminate(self): prog = os.path.join(os.path.dirname(__file__), 'echo.py') @@ -1635,6 +1639,7 @@ transp.terminate() self.loop.run_until_complete(proto.completed) self.check_terminated(proto.returncode) + transp.close() @unittest.skipIf(sys.platform == 'win32', "Don't have SIGHUP") def test_subprocess_send_signal(self): @@ -1650,6 +1655,7 @@ transp.send_signal(signal.SIGHUP) self.loop.run_until_complete(proto.completed) self.assertEqual(-signal.SIGHUP, proto.returncode) + transp.close() def test_subprocess_stderr(self): prog = os.path.join(os.path.dirname(__file__), 'echo2.py') @@ -1784,6 +1790,22 @@ def create_event_loop(self): return asyncio.ProactorEventLoop() + if not sslproto._is_sslproto_available(): + def test_create_ssl_connection(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + + def test_create_server_ssl(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + + def test_create_server_ssl_verify_failed(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + + def test_create_server_ssl_match_failed(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + + def test_create_server_ssl_verified(self): + raise unittest.SkipTest("need python 3.5 (ssl.MemoryBIO)") + def test_legacy_create_ssl_connection(self): raise unittest.SkipTest("IocpEventLoop incompatible with legacy SSL") diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py --- a/Lib/test/test_asyncio/test_futures.py +++ b/Lib/test/test_asyncio/test_futures.py @@ -29,6 +29,7 @@ def setUp(self): self.loop = self.new_test_loop() + self.addCleanup(self.loop.close) def test_initial_state(self): f = asyncio.Future(loop=self.loop) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1744,6 +1744,7 @@ test_utils.MockPattern( 'Fatal error on transport\nprotocol:.*\ntransport:.*'), exc_info=(ConnectionRefusedError, MOCK_ANY, MOCK_ANY)) + transport.close() if __name__ == '__main__': diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -598,6 +598,7 @@ # This is a bit overspecified. :-( m_log.warning.assert_called_with( 'pipe closed by peer or os.write(pipe, data) raised exception.') + tr.close() @mock.patch('os.write') def test_write_close(self, m_write): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 05:05:10 2015 From: python-checkins at python.org (larry.hastings) Date: Thu, 15 Jan 2015 04:05:10 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Added_schedule_for_3=2E4=2E3_?= =?utf-8?q?rc1_and_final=2E__Revised_schedule_for_3=2E5=2E0a1=2E?= Message-ID: <20150115040500.8767.49747@psf.io> https://hg.python.org/peps/rev/990461fae87a changeset: 5672:990461fae87a user: Larry Hastings date: Wed Jan 14 23:04:57 2015 -0500 summary: Added schedule for 3.4.3 rc1 and final. Revised schedule for 3.5.0a1. files: pep-0429.txt | 6 ++++++ pep-0478.txt | 2 +- 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/pep-0429.txt b/pep-0429.txt --- a/pep-0429.txt +++ b/pep-0429.txt @@ -62,6 +62,12 @@ - 3.4.2 candidate 1: September 22, 2014 - 3.4.2 final: October 6, 2014 +3.4.3 schedule +-------------- + +- 3.4.3 candidate 1: February 8, 2015 +- 3.4.3 final: February 22, 2015 + Features for 3.4 diff --git a/pep-0478.txt b/pep-0478.txt --- a/pep-0478.txt +++ b/pep-0478.txt @@ -36,7 +36,7 @@ The releases: -- 3.5.0 alpha 1: February 1, 2015 +- 3.5.0 alpha 1: February 8, 2015 - 3.5.0 alpha 2: March 8, 2015 - 3.5.0 alpha 3: March 28, 2015 - 3.5.0 alpha 4: April 19, 2015 -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Thu Jan 15 06:00:54 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 15 Jan 2015 05:00:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMwNjMp?= Message-ID: <20150115050026.72567.9075@psf.io> https://hg.python.org/cpython/rev/731a36c13629 changeset: 94148:731a36c13629 parent: 94145:61a045ac0006 parent: 94146:db09d760b965 user: Benjamin Peterson date: Thu Jan 15 00:00:16 2015 -0500 summary: merge 3.4 (#23063) files: Lib/distutils/command/check.py | 8 ++-- Lib/distutils/tests/test_check.py | 31 +++++++++++++++++++ Misc/NEWS | 3 + 3 files changed, 38 insertions(+), 4 deletions(-) diff --git a/Lib/distutils/command/check.py b/Lib/distutils/command/check.py --- a/Lib/distutils/command/check.py +++ b/Lib/distutils/command/check.py @@ -122,7 +122,7 @@ """Returns warnings when the provided data doesn't compile.""" source_path = StringIO() parser = Parser() - settings = frontend.OptionParser().get_default_values() + settings = frontend.OptionParser(components=(Parser,)).get_default_values() settings.tab_width = 4 settings.pep_references = None settings.rfc_references = None @@ -138,8 +138,8 @@ document.note_source(source_path, -1) try: parser.parse(data, document) - except AttributeError: - reporter.messages.append((-1, 'Could not finish the parsing.', - '', {})) + except AttributeError as e: + reporter.messages.append( + (-1, 'Could not finish the parsing: %s.' % e, '', {})) return reporter.messages diff --git a/Lib/distutils/tests/test_check.py b/Lib/distutils/tests/test_check.py --- a/Lib/distutils/tests/test_check.py +++ b/Lib/distutils/tests/test_check.py @@ -1,4 +1,5 @@ """Tests for distutils.command.check.""" +import textwrap import unittest from test.support import run_unittest @@ -92,6 +93,36 @@ cmd = self._run(metadata, strict=1, restructuredtext=1) self.assertEqual(cmd._warnings, 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") + def test_check_restructuredtext_with_syntax_highlight(self): + # Don't fail if there is a `code` or `code-block` directive + + example_rst_docs = [] + example_rst_docs.append(textwrap.dedent("""\ + Here's some code: + + .. code:: python + + def foo(): + pass + """)) + example_rst_docs.append(textwrap.dedent("""\ + Here's some code: + + .. code-block:: python + + def foo(): + pass + """)) + + for rest_with_code in example_rst_docs: + pkg_info, dist = self.create_dist(long_description=rest_with_code) + cmd = check(dist) + cmd.check_restructuredtext() + self.assertEqual(cmd._warnings, 0) + msgs = cmd._check_rst_data(rest_with_code) + self.assertEqual(len(msgs), 0) + def test_check_all(self): metadata = {'url': 'xxx', 'author': 'xxx'} diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,9 @@ Library ------- +- Issue #23063: In the disutils' check command, fix parsing of reST with code or + code-block directives. + - Issue #23209, #23225: selectors.BaseSelector.get_key() now raises a RuntimeError if the selector is closed. And selectors.BaseSelector.close() now clears its internal reference to the selector mapping to break a -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 06:00:54 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 15 Jan 2015 05:00:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_fix_parsing_re?= =?utf-8?q?ST_with_code_or_code-block_directives_=28closes_=2323063=29?= Message-ID: <20150115050025.11561.28807@psf.io> https://hg.python.org/cpython/rev/db09d760b965 changeset: 94146:db09d760b965 branch: 3.4 parent: 94144:463bbd862887 user: Benjamin Peterson date: Wed Jan 14 23:56:35 2015 -0500 summary: fix parsing reST with code or code-block directives (closes #23063) Patch by Marc Abramowitz. files: Lib/distutils/command/check.py | 8 ++-- Lib/distutils/tests/test_check.py | 31 +++++++++++++++++++ Misc/NEWS | 3 + 3 files changed, 38 insertions(+), 4 deletions(-) diff --git a/Lib/distutils/command/check.py b/Lib/distutils/command/check.py --- a/Lib/distutils/command/check.py +++ b/Lib/distutils/command/check.py @@ -122,7 +122,7 @@ """Returns warnings when the provided data doesn't compile.""" source_path = StringIO() parser = Parser() - settings = frontend.OptionParser().get_default_values() + settings = frontend.OptionParser(components=(Parser,)).get_default_values() settings.tab_width = 4 settings.pep_references = None settings.rfc_references = None @@ -138,8 +138,8 @@ document.note_source(source_path, -1) try: parser.parse(data, document) - except AttributeError: - reporter.messages.append((-1, 'Could not finish the parsing.', - '', {})) + except AttributeError as e: + reporter.messages.append( + (-1, 'Could not finish the parsing: %s.' % e, '', {})) return reporter.messages diff --git a/Lib/distutils/tests/test_check.py b/Lib/distutils/tests/test_check.py --- a/Lib/distutils/tests/test_check.py +++ b/Lib/distutils/tests/test_check.py @@ -1,4 +1,5 @@ """Tests for distutils.command.check.""" +import textwrap import unittest from test.support import run_unittest @@ -92,6 +93,36 @@ cmd = self._run(metadata, strict=1, restructuredtext=1) self.assertEqual(cmd._warnings, 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") + def test_check_restructuredtext_with_syntax_highlight(self): + # Don't fail if there is a `code` or `code-block` directive + + example_rst_docs = [] + example_rst_docs.append(textwrap.dedent("""\ + Here's some code: + + .. code:: python + + def foo(): + pass + """)) + example_rst_docs.append(textwrap.dedent("""\ + Here's some code: + + .. code-block:: python + + def foo(): + pass + """)) + + for rest_with_code in example_rst_docs: + pkg_info, dist = self.create_dist(long_description=rest_with_code) + cmd = check(dist) + cmd.check_restructuredtext() + self.assertEqual(cmd._warnings, 0) + msgs = cmd._check_rst_data(rest_with_code) + self.assertEqual(len(msgs), 0) + def test_check_all(self): metadata = {'url': 'xxx', 'author': 'xxx'} diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #23063: In the disutils' check command, fix parsing of reST with code or + code-block directives. + - Issue #23209, #23225: selectors.BaseSelector.close() now clears its internal reference to the selector mapping to break a reference cycle. Initial patch written by Martin Richard. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 06:00:54 2015 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 15 Jan 2015 05:00:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_fix_parsing_re?= =?utf-8?q?ST_with_code_or_code-block_directives_=28closes_=2323063=29?= Message-ID: <20150115050026.11591.28294@psf.io> https://hg.python.org/cpython/rev/38826e21f0db changeset: 94147:38826e21f0db branch: 2.7 parent: 94137:e280a04625cc user: Benjamin Peterson date: Wed Jan 14 23:56:35 2015 -0500 summary: fix parsing reST with code or code-block directives (closes #23063) Patch by Marc Abramowitz. files: Lib/distutils/command/check.py | 8 ++-- Lib/distutils/tests/test_check.py | 31 +++++++++++++++++++ Misc/NEWS | 3 + 3 files changed, 38 insertions(+), 4 deletions(-) diff --git a/Lib/distutils/command/check.py b/Lib/distutils/command/check.py --- a/Lib/distutils/command/check.py +++ b/Lib/distutils/command/check.py @@ -126,7 +126,7 @@ """Returns warnings when the provided data doesn't compile.""" source_path = StringIO() parser = Parser() - settings = frontend.OptionParser().get_default_values() + settings = frontend.OptionParser(components=(Parser,)).get_default_values() settings.tab_width = 4 settings.pep_references = None settings.rfc_references = None @@ -142,8 +142,8 @@ document.note_source(source_path, -1) try: parser.parse(data, document) - except AttributeError: - reporter.messages.append((-1, 'Could not finish the parsing.', - '', {})) + except AttributeError as e: + reporter.messages.append( + (-1, 'Could not finish the parsing: %s.' % e, '', {})) return reporter.messages diff --git a/Lib/distutils/tests/test_check.py b/Lib/distutils/tests/test_check.py --- a/Lib/distutils/tests/test_check.py +++ b/Lib/distutils/tests/test_check.py @@ -1,5 +1,6 @@ # -*- encoding: utf8 -*- """Tests for distutils.command.check.""" +import textwrap import unittest from test.test_support import run_unittest @@ -93,6 +94,36 @@ cmd = self._run(metadata, strict=1, restructuredtext=1) self.assertEqual(cmd._warnings, 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") + def test_check_restructuredtext_with_syntax_highlight(self): + # Don't fail if there is a `code` or `code-block` directive + + example_rst_docs = [] + example_rst_docs.append(textwrap.dedent("""\ + Here's some code: + + .. code:: python + + def foo(): + pass + """)) + example_rst_docs.append(textwrap.dedent("""\ + Here's some code: + + .. code-block:: python + + def foo(): + pass + """)) + + for rest_with_code in example_rst_docs: + pkg_info, dist = self.create_dist(long_description=rest_with_code) + cmd = check(dist) + cmd.check_restructuredtext() + self.assertEqual(cmd._warnings, 0) + msgs = cmd._check_rst_data(rest_with_code) + self.assertEqual(len(msgs), 0) + def test_check_all(self): metadata = {'url': 'xxx', 'author': 'xxx'} diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,9 @@ Library ------- +- Issue #23063: In the disutils' check command, fix parsing of reST with code or + code-block directives. + - Issue #21356: Make ssl.RAND_egd() optional to support LibreSSL. The availability of the function is checked during the compilation. Patch written by Bernard Spil. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 06:58:10 2015 From: python-checkins at python.org (ethan.furman) Date: Thu, 15 Jan 2015 05:58:10 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUyMDQ2Nzog?= =?utf-8?b?Y2xhcmlmeSBfX2luaXRfXydzIHJvbGU=?= Message-ID: <20150115055754.125886.49389@psf.io> https://hg.python.org/cpython/rev/94696e457461 changeset: 94149:94696e457461 branch: 3.3 parent: 94019:d1af6f3a8ce3 user: Ethan Furman date: Wed Jan 14 21:56:10 2015 -0800 summary: Issue20467: clarify __init__'s role files: Doc/reference/datamodel.rst | 18 +++++++++++------- 1 files changed, 11 insertions(+), 7 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1081,13 +1081,17 @@ .. index:: pair: class; constructor - Called when the instance is created. The arguments are those passed to the - class constructor expression. If a base class has an :meth:`__init__` method, - the derived class's :meth:`__init__` method, if any, must explicitly call it to - ensure proper initialization of the base class part of the instance; for - example: ``BaseClass.__init__(self, [args...])``. As a special constraint on - constructors, no value may be returned; doing so will cause a :exc:`TypeError` - to be raised at runtime. + Called after the instance has been created (by :meth:`__new__`), but before + it is returned to the caller. The arguments are those passed to the + class constructor expression. If a base class has an :meth:`__init__` + method, the derived class's :meth:`__init__` method, if any, must explicitly + call it to ensure proper initialization of the base class part of the + instance; for example: ``BaseClass.__init__(self, [args...])``. + + Because :meth:`__new__` and :meth:`__init__` work together in constructing + objects (:meth:`__new__` to create it, and :meth:`__init__` to customise it), + no non-``None`` value may be returned by :meth:`__init__`; doing so will + cause a :exc:`TypeError` to be raised at runtime. .. method:: object.__del__(self) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 06:58:10 2015 From: python-checkins at python.org (ethan.furman) Date: Thu, 15 Jan 2015 05:58:10 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_Issue20467=3A_clarify_=5F=5Finit=5F=5F=27s_role?= Message-ID: <20150115055754.11589.41599@psf.io> https://hg.python.org/cpython/rev/46c9b34f31b8 changeset: 94150:46c9b34f31b8 branch: 3.4 parent: 94146:db09d760b965 parent: 94149:94696e457461 user: Ethan Furman date: Wed Jan 14 21:56:49 2015 -0800 summary: Issue20467: clarify __init__'s role files: Doc/reference/datamodel.rst | 18 +++++++++++------- 1 files changed, 11 insertions(+), 7 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1100,13 +1100,17 @@ .. index:: pair: class; constructor - Called when the instance is created. The arguments are those passed to the - class constructor expression. If a base class has an :meth:`__init__` method, - the derived class's :meth:`__init__` method, if any, must explicitly call it to - ensure proper initialization of the base class part of the instance; for - example: ``BaseClass.__init__(self, [args...])``. As a special constraint on - constructors, no value may be returned; doing so will cause a :exc:`TypeError` - to be raised at runtime. + Called after the instance has been created (by :meth:`__new__`), but before + it is returned to the caller. The arguments are those passed to the + class constructor expression. If a base class has an :meth:`__init__` + method, the derived class's :meth:`__init__` method, if any, must explicitly + call it to ensure proper initialization of the base class part of the + instance; for example: ``BaseClass.__init__(self, [args...])``. + + Because :meth:`__new__` and :meth:`__init__` work together in constructing + objects (:meth:`__new__` to create it, and :meth:`__init__` to customise it), + no non-``None`` value may be returned by :meth:`__init__`; doing so will + cause a :exc:`TypeError` to be raised at runtime. .. method:: object.__del__(self) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 06:58:10 2015 From: python-checkins at python.org (ethan.furman) Date: Thu, 15 Jan 2015 05:58:10 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogSXNzdWUyMDQ2NzogY2xhcmlmeSBfX2luaXRfXydzIHJvbGU=?= Message-ID: <20150115055755.125888.66317@psf.io> https://hg.python.org/cpython/rev/d0421de8ee11 changeset: 94151:d0421de8ee11 parent: 94148:731a36c13629 parent: 94150:46c9b34f31b8 user: Ethan Furman date: Wed Jan 14 21:57:15 2015 -0800 summary: Issue20467: clarify __init__'s role files: Doc/reference/datamodel.rst | 18 +++++++++++------- 1 files changed, 11 insertions(+), 7 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1100,13 +1100,17 @@ .. index:: pair: class; constructor - Called when the instance is created. The arguments are those passed to the - class constructor expression. If a base class has an :meth:`__init__` method, - the derived class's :meth:`__init__` method, if any, must explicitly call it to - ensure proper initialization of the base class part of the instance; for - example: ``BaseClass.__init__(self, [args...])``. As a special constraint on - constructors, no value may be returned; doing so will cause a :exc:`TypeError` - to be raised at runtime. + Called after the instance has been created (by :meth:`__new__`), but before + it is returned to the caller. The arguments are those passed to the + class constructor expression. If a base class has an :meth:`__init__` + method, the derived class's :meth:`__init__` method, if any, must explicitly + call it to ensure proper initialization of the base class part of the + instance; for example: ``BaseClass.__init__(self, [args...])``. + + Because :meth:`__new__` and :meth:`__init__` work together in constructing + objects (:meth:`__new__` to create it, and :meth:`__init__` to customise it), + no non-``None`` value may be returned by :meth:`__init__`; doing so will + cause a :exc:`TypeError` to be raised at runtime. .. method:: object.__del__(self) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 07:02:41 2015 From: python-checkins at python.org (ethan.furman) Date: Thu, 15 Jan 2015 06:02:41 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUyMDQ2Nzog?= =?utf-8?b?Y2xhcmlmeSBfX2luaXRfXydzIHJvbGU=?= Message-ID: <20150115060235.22405.49428@psf.io> https://hg.python.org/cpython/rev/7972b18b6e42 changeset: 94152:7972b18b6e42 branch: 2.7 parent: 94147:38826e21f0db user: Ethan Furman date: Wed Jan 14 22:02:10 2015 -0800 summary: Issue20467: clarify __init__'s role files: Doc/reference/datamodel.rst | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1225,13 +1225,17 @@ .. index:: pair: class; constructor - Called when the instance is created. The arguments are those passed to the + Called after the instance has been created (by :meth:`__new__`), but before + it is returned to the caller. The arguments are those passed to the class constructor expression. If a base class has an :meth:`__init__` method, the derived class's :meth:`__init__` method, if any, must explicitly call it to ensure proper initialization of the base class part of the instance; for - example: ``BaseClass.__init__(self, [args...])``. As a special constraint on - constructors, no value may be returned; doing so will cause a :exc:`TypeError` - to be raised at runtime. + example: ``BaseClass.__init__(self, [args...])``. + + Because :meth:`__new__` and :meth:`__init__` work together in constructing + objects (:meth:`__new__` to create it, and :meth:`__init__` to customise it), + no non-``None`` value may be returned by :meth:`__init__`; doing so will + cause a :exc:`TypeError` to be raised at runtime. .. method:: object.__del__(self) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 07:26:34 2015 From: python-checkins at python.org (ethan.furman) Date: Thu, 15 Jan 2015 06:26:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue22988=3A_clarify_yield_and_exception_blocks?= Message-ID: <20150115062629.125896.26217@psf.io> https://hg.python.org/cpython/rev/55ba19d084fb changeset: 94154:55ba19d084fb parent: 94151:d0421de8ee11 parent: 94153:0e48b0908651 user: Ethan Furman date: Wed Jan 14 22:26:04 2015 -0800 summary: Issue22988: clarify yield and exception blocks files: Doc/reference/expressions.rst | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -330,8 +330,9 @@ time, the execution proceeds to the first yield expression, where it is suspended again, returning the value of :token:`expression_list` to the generator's caller. By suspended, we mean that all local state is retained, including the -current bindings of local variables, the instruction pointer, and the internal -evaluation stack. When the execution is resumed by calling one of the +current bindings of local variables, the instruction pointer, the internal +evaluation stack, and the state of any exception handling. When the execution +is resumed by calling one of the generator's methods, the function can proceed exactly as if the yield expression were just another external call. The value of the yield expression after resuming depends on the method which resumed the execution. If @@ -348,8 +349,8 @@ where the execution should continue after it yields; the control is always transferred to the generator's caller. -Yield expressions are allowed in the :keyword:`try` clause of a :keyword:`try` -... :keyword:`finally` construct. If the generator is not resumed before it is +Yield expressions are allowed anywhere in a :keyword:`try` construct. If the +generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator's :meth:`~generator.close` method will be called, allowing any pending :keyword:`finally` clauses to execute. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 07:26:34 2015 From: python-checkins at python.org (ethan.furman) Date: Thu, 15 Jan 2015 06:26:34 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUyMjk4ODog?= =?utf-8?q?clarify_yield_and_exception_blocks?= Message-ID: <20150115062629.22403.36139@psf.io> https://hg.python.org/cpython/rev/0e48b0908651 changeset: 94153:0e48b0908651 branch: 3.4 parent: 94150:46c9b34f31b8 user: Ethan Furman date: Wed Jan 14 22:25:27 2015 -0800 summary: Issue22988: clarify yield and exception blocks files: Doc/reference/expressions.rst | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -330,8 +330,9 @@ time, the execution proceeds to the first yield expression, where it is suspended again, returning the value of :token:`expression_list` to the generator's caller. By suspended, we mean that all local state is retained, including the -current bindings of local variables, the instruction pointer, and the internal -evaluation stack. When the execution is resumed by calling one of the +current bindings of local variables, the instruction pointer, the internal +evaluation stack, and the state of any exception handling. When the execution +is resumed by calling one of the generator's methods, the function can proceed exactly as if the yield expression were just another external call. The value of the yield expression after resuming depends on the method which resumed the execution. If @@ -348,8 +349,8 @@ where the execution should continue after it yields; the control is always transferred to the generator's caller. -Yield expressions are allowed in the :keyword:`try` clause of a :keyword:`try` -... :keyword:`finally` construct. If the generator is not resumed before it is +Yield expressions are allowed anywhere in a :keyword:`try` construct. If the +generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator's :meth:`~generator.close` method will be called, allowing any pending :keyword:`finally` clauses to execute. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 07:32:54 2015 From: python-checkins at python.org (ethan.furman) Date: Thu, 15 Jan 2015 06:32:54 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUyMjk5Nzog?= =?utf-8?q?minor_doc_update=3B_thanks_to_Simoen_Visser?= Message-ID: <20150115063254.11571.86166@psf.io> https://hg.python.org/cpython/rev/522d13d8e76e changeset: 94155:522d13d8e76e branch: 3.4 parent: 94153:0e48b0908651 user: Ethan Furman date: Wed Jan 14 22:31:50 2015 -0800 summary: Issue22997: minor doc update; thanks to Simoen Visser files: Doc/library/enum.rst | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -404,7 +404,7 @@ new class derived from :class:`Enum` is returned. In other words, the above assignment to :class:`Animal` is equivalent to:: - >>> class Animals(Enum): + >>> class Animal(Enum): ... ant = 1 ... bee = 2 ... cat = 3 @@ -421,7 +421,7 @@ function in separate module, and also may not work on IronPython or Jython). The solution is to specify the module name explicitly as follows:: - >>> Animals = Enum('Animals', 'ant bee cat dog', module=__name__) + >>> Animal = Enum('Animal', 'ant bee cat dog', module=__name__) .. warning:: @@ -434,7 +434,7 @@ to find the class. For example, if the class was made available in class SomeData in the global scope:: - >>> Animals = Enum('Animals', 'ant bee cat dog', qualname='SomeData.Animals') + >>> Animal = Enum('Animal', 'ant bee cat dog', qualname='SomeData.Animal') The complete signature is:: @@ -447,6 +447,10 @@ 'red green blue' | 'red,green,blue' | 'red, green, blue' + or an iterator of names:: + + ['red', 'green', 'blue'] + or an iterator of (name, value) pairs:: [('cyan', 4), ('magenta', 5), ('yellow', 6)] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 07:32:54 2015 From: python-checkins at python.org (ethan.furman) Date: Thu, 15 Jan 2015 06:32:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue22997=3A_minor_doc_update=3B_thanks_to_Simoen_Visse?= =?utf-8?q?r?= Message-ID: <20150115063254.11579.95109@psf.io> https://hg.python.org/cpython/rev/31ae2dcaaed8 changeset: 94156:31ae2dcaaed8 parent: 94154:55ba19d084fb parent: 94155:522d13d8e76e user: Ethan Furman date: Wed Jan 14 22:32:29 2015 -0800 summary: Issue22997: minor doc update; thanks to Simoen Visser files: Doc/library/enum.rst | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -405,7 +405,7 @@ new class derived from :class:`Enum` is returned. In other words, the above assignment to :class:`Animal` is equivalent to:: - >>> class Animals(Enum): + >>> class Animal(Enum): ... ant = 1 ... bee = 2 ... cat = 3 @@ -422,7 +422,7 @@ function in separate module, and also may not work on IronPython or Jython). The solution is to specify the module name explicitly as follows:: - >>> Animals = Enum('Animals', 'ant bee cat dog', module=__name__) + >>> Animal = Enum('Animal', 'ant bee cat dog', module=__name__) .. warning:: @@ -435,7 +435,7 @@ to find the class. For example, if the class was made available in class SomeData in the global scope:: - >>> Animals = Enum('Animals', 'ant bee cat dog', qualname='SomeData.Animals') + >>> Animal = Enum('Animal', 'ant bee cat dog', qualname='SomeData.Animal') The complete signature is:: @@ -448,6 +448,10 @@ 'red green blue' | 'red,green,blue' | 'red, green, blue' + or an iterator of names:: + + ['red', 'green', 'blue'] + or an iterator of (name, value) pairs:: [('cyan', 4), ('magenta', 5), ('yellow', 6)] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 08:16:48 2015 From: python-checkins at python.org (georg.brandl) Date: Thu, 15 Jan 2015 07:16:48 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMzI0?= =?utf-8?q?4=3A_fix_typo=2E_Thanks_Mayank_Tripathi_for_the_patch=2E?= Message-ID: <20150115071633.8755.99201@psf.io> https://hg.python.org/cpython/rev/3e4d4c4968bb changeset: 94158:3e4d4c4968bb branch: 3.4 parent: 94155:522d13d8e76e user: Georg Brandl date: Thu Jan 15 08:16:01 2015 +0100 summary: Closes #23244: fix typo. Thanks Mayank Tripathi for the patch. files: Doc/glossary.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -292,7 +292,7 @@ generator A function which returns an iterator. It looks like a normal function except that it contains :keyword:`yield` statements for producing a series - a values usable in a for-loop or that can be retrieved one at a time with + of values usable in a for-loop or that can be retrieved one at a time with the :func:`next` function. Each :keyword:`yield` temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator resumes, it -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 08:16:50 2015 From: python-checkins at python.org (georg.brandl) Date: Thu, 15 Jan 2015 07:16:50 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E4?= Message-ID: <20150115071634.72577.82979@psf.io> https://hg.python.org/cpython/rev/2a8378e69d75 changeset: 94159:2a8378e69d75 parent: 94156:31ae2dcaaed8 parent: 94158:3e4d4c4968bb user: Georg Brandl date: Thu Jan 15 08:16:25 2015 +0100 summary: merge with 3.4 files: Doc/glossary.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -292,7 +292,7 @@ generator A function which returns an iterator. It looks like a normal function except that it contains :keyword:`yield` statements for producing a series - a values usable in a for-loop or that can be retrieved one at a time with + of values usable in a for-loop or that can be retrieved one at a time with the :func:`next` function. Each :keyword:`yield` temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator resumes, it -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 08:16:50 2015 From: python-checkins at python.org (georg.brandl) Date: Thu, 15 Jan 2015 07:16:50 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMyMzI0?= =?utf-8?q?4=3A_fix_typo=2E_Thanks_Mayank_Tripathi_for_the_patch=2E?= Message-ID: <20150115071633.9920.54554@psf.io> https://hg.python.org/cpython/rev/25a1ce2a6f9b changeset: 94157:25a1ce2a6f9b branch: 2.7 parent: 94152:7972b18b6e42 user: Georg Brandl date: Thu Jan 15 08:16:01 2015 +0100 summary: Closes #23244: fix typo. Thanks Mayank Tripathi for the patch. files: Doc/glossary.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -284,7 +284,7 @@ generator A function which returns an iterator. It looks like a normal function except that it contains :keyword:`yield` statements for producing a series - a values usable in a for-loop or that can be retrieved one at a time with + of values usable in a for-loop or that can be retrieved one at a time with the :func:`next` function. Each :keyword:`yield` temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator resumes, it -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Thu Jan 15 08:37:30 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 15 Jan 2015 08:37:30 +0100 Subject: [Python-checkins] Daily reference leaks (61a045ac0006): sum=-6 Message-ID: results for 61a045ac0006 on branch "default" -------------------------------------------- test_collections leaked [-2, -4, 0] references, sum=-6 test_collections leaked [-1, -2, 0] memory blocks, sum=-3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogDncN0A', '-x'] From python-checkins at python.org Thu Jan 15 09:36:31 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 08:36:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_StreamWriter?= =?utf-8?q?=3A_close=28=29_now_clears_the_reference_to_the_transport?= Message-ID: <20150115083613.8765.13586@psf.io> https://hg.python.org/cpython/rev/6ab2575bc12b changeset: 94160:6ab2575bc12b branch: 3.4 parent: 94158:3e4d4c4968bb user: Victor Stinner date: Thu Jan 15 09:33:50 2015 +0100 summary: StreamWriter: close() now clears the reference to the transport StreamWriter now raises an exception if it is closed: write(), writelines(), write_eof(), can_write_eof(), get_extra_info(), drain(). files: Lib/asyncio/streams.py | 25 +++++++++++++++++++++---- 1 files changed, 21 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -258,8 +258,22 @@ self._reader = reader self._loop = loop + def close(self): + if self._transport is None: + return + self._transport.close() + self._transport = None + + def _check_closed(self): + if self._transport is None: + raise RuntimeError('StreamWriter is closed') + def __repr__(self): - info = [self.__class__.__name__, 'transport=%r' % self._transport] + info = [self.__class__.__name__] + if self._transport is not None: + info.append('transport=%r' % self._transport) + else: + info.append('closed') if self._reader is not None: info.append('reader=%r' % self._reader) return '<%s>' % ' '.join(info) @@ -269,21 +283,23 @@ return self._transport def write(self, data): + self._check_closed() self._transport.write(data) def writelines(self, data): + self._check_closed() self._transport.writelines(data) def write_eof(self): + self._check_closed() return self._transport.write_eof() def can_write_eof(self): + self._check_closed() return self._transport.can_write_eof() - def close(self): - return self._transport.close() - def get_extra_info(self, name, default=None): + self._check_closed() return self._transport.get_extra_info(name, default) @coroutine @@ -295,6 +311,7 @@ w.write(data) yield from w.drain() """ + self._check_closed() if self._reader is not None: exc = self._reader.exception() if exc is not None: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 09:36:31 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 08:36:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150115083614.11587.29110@psf.io> https://hg.python.org/cpython/rev/681e54e6d11e changeset: 94161:681e54e6d11e parent: 94159:2a8378e69d75 parent: 94160:6ab2575bc12b user: Victor Stinner date: Thu Jan 15 09:35:29 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/streams.py | 25 +++++++++++++++++++++---- 1 files changed, 21 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -258,8 +258,22 @@ self._reader = reader self._loop = loop + def close(self): + if self._transport is None: + return + self._transport.close() + self._transport = None + + def _check_closed(self): + if self._transport is None: + raise RuntimeError('StreamWriter is closed') + def __repr__(self): - info = [self.__class__.__name__, 'transport=%r' % self._transport] + info = [self.__class__.__name__] + if self._transport is not None: + info.append('transport=%r' % self._transport) + else: + info.append('closed') if self._reader is not None: info.append('reader=%r' % self._reader) return '<%s>' % ' '.join(info) @@ -269,21 +283,23 @@ return self._transport def write(self, data): + self._check_closed() self._transport.write(data) def writelines(self, data): + self._check_closed() self._transport.writelines(data) def write_eof(self): + self._check_closed() return self._transport.write_eof() def can_write_eof(self): + self._check_closed() return self._transport.can_write_eof() - def close(self): - return self._transport.close() - def get_extra_info(self, name, default=None): + self._check_closed() return self._transport.get_extra_info(name, default) @coroutine @@ -295,6 +311,7 @@ w.write(data) yield from w.drain() """ + self._check_closed() if self._reader is not None: exc = self._reader.exception() if exc is not None: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 09:42:26 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 08:42:26 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNTYw?= =?utf-8?q?=3A_Fix_SSLProtocol=2E=5Fon=5Fhandshake=5Fcomplete=28=29?= Message-ID: <20150115084223.22397.77030@psf.io> https://hg.python.org/cpython/rev/fb3761de0d3c changeset: 94162:fb3761de0d3c branch: 3.4 parent: 94160:6ab2575bc12b user: Victor Stinner date: Thu Jan 15 09:41:48 2015 +0100 summary: Issue #22560: Fix SSLProtocol._on_handshake_complete() Don't call immediatly self._process_write_backlog() but schedule the call using call_soon(). _on_handshake_complete() can be called indirectly from _process_write_backlog(), and _process_write_backlog() is not reentrant. files: Lib/asyncio/sslproto.py | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -572,8 +572,12 @@ # wait until protocol.connection_made() has been called self._waiter._set_result_unless_cancelled(None) self._session_established = True - # In case transport.write() was already called - self._process_write_backlog() + # In case transport.write() was already called. Don't call + # immediatly _process_write_backlog(), but schedule it: + # _on_handshake_complete() can be called indirectly from + # _process_write_backlog(), and _process_write_backlog() is not + # reentrant. + self._loop.call(self._process_write_backlog) def _process_write_backlog(self): # Try to make progress on the write backlog. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 09:45:25 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 08:45:25 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNTYw?= =?utf-8?q?=3A_Fix_typo=3A_call_-=3E_call=5Fsoon?= Message-ID: <20150115084523.72571.30858@psf.io> https://hg.python.org/cpython/rev/3c37825d85d3 changeset: 94163:3c37825d85d3 branch: 3.4 user: Victor Stinner date: Thu Jan 15 09:44:13 2015 +0100 summary: Issue #22560: Fix typo: call -> call_soon files: Lib/asyncio/sslproto.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -577,7 +577,7 @@ # _on_handshake_complete() can be called indirectly from # _process_write_backlog(), and _process_write_backlog() is not # reentrant. - self._loop.call(self._process_write_backlog) + self._loop.call_soon(self._process_write_backlog) def _process_write_backlog(self): # Try to make progress on the write backlog. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 09:45:25 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 08:45:25 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150115084524.8749.36976@psf.io> https://hg.python.org/cpython/rev/69285bf7e865 changeset: 94164:69285bf7e865 parent: 94161:681e54e6d11e parent: 94163:3c37825d85d3 user: Victor Stinner date: Thu Jan 15 09:44:24 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/sslproto.py | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -572,8 +572,12 @@ # wait until protocol.connection_made() has been called self._waiter._set_result_unless_cancelled(None) self._session_established = True - # In case transport.write() was already called - self._process_write_backlog() + # In case transport.write() was already called. Don't call + # immediatly _process_write_backlog(), but schedule it: + # _on_handshake_complete() can be called indirectly from + # _process_write_backlog(), and _process_write_backlog() is not + # reentrant. + self._loop.call_soon(self._process_write_backlog) def _process_write_backlog(self): # Try to make progress on the write backlog. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:31:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:31:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogU1NMUHJvdG9jb2w6?= =?utf-8?q?_set_the_=5Ftransport_attribute_in_the_constructor?= Message-ID: <20150115122510.22407.55194@psf.io> https://hg.python.org/cpython/rev/1fca25b4ea1f changeset: 94166:1fca25b4ea1f branch: 3.4 user: Victor Stinner date: Thu Jan 15 13:16:27 2015 +0100 summary: SSLProtocol: set the _transport attribute in the constructor files: Lib/asyncio/sslproto.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -417,6 +417,7 @@ self._session_established = False self._in_handshake = False self._in_shutdown = False + self._transport = None def connection_made(self, transport): """Called when the low-level connection is made. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:31:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:31:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjQz?= =?utf-8?q?=3A_Fix_asyncio=2E=5FUnixWritePipeTransport=2Eclose=28=29?= Message-ID: <20150115122510.72559.64359@psf.io> https://hg.python.org/cpython/rev/2f13d53f4680 changeset: 94167:2f13d53f4680 branch: 3.4 user: Victor Stinner date: Thu Jan 15 13:16:50 2015 +0100 summary: Issue #23243: Fix asyncio._UnixWritePipeTransport.close() Do nothing if the transport is already closed. Before it was not possible to close the transport twice. files: Lib/asyncio/unix_events.py | 2 +- Lib/test/test_asyncio/test_unix_events.py | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -516,7 +516,7 @@ self._loop.call_soon(self._call_connection_lost, None) def close(self): - if not self._closing: + if self._pipe is not None and not self._closing: # write_eof is all what we needed to close the write pipe self.write_eof() diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -766,6 +766,9 @@ tr.close() tr.write_eof.assert_called_with() + # closing the transport twice must not fail + tr.close() + def test_close_closing(self): tr = unix_events._UnixWritePipeTransport( self.loop, self.pipe, self.protocol) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:31:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:31:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjQz?= =?utf-8?q?=3A_Close_explicitly_event_loops_in_asyncio_tests?= Message-ID: <20150115122511.11591.85912@psf.io> https://hg.python.org/cpython/rev/aef0f9b4e729 changeset: 94168:aef0f9b4e729 branch: 3.4 user: Victor Stinner date: Thu Jan 15 13:17:34 2015 +0100 summary: Issue #23243: Close explicitly event loops in asyncio tests files: Lib/test/test_asyncio/test_base_events.py | 1 + Lib/test/test_asyncio/test_proactor_events.py | 4 ++ Lib/test/test_asyncio/test_selector_events.py | 16 +++++++++- 3 files changed, 20 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -590,6 +590,7 @@ raise ValueError('spam') loop = Loop() + self.addCleanup(loop.close) asyncio.set_event_loop(loop) def run_loop(): diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -16,6 +16,7 @@ def setUp(self): self.loop = self.new_test_loop() + self.addCleanup(self.loop.close) self.proactor = mock.Mock() self.loop._proactor = self.proactor self.protocol = test_utils.make_test_protocol(asyncio.Protocol) @@ -459,6 +460,9 @@ self.assertIsNone(self.loop._ssock) self.assertIsNone(self.loop._csock) + # Don't call close(): _close_self_pipe() cannot be called twice + self.loop._closed = True + def test_close(self): self.loop._close_self_pipe = mock.Mock() self.loop.close() diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -24,6 +24,11 @@ class TestBaseSelectorEventLoop(BaseSelectorEventLoop): + def close(self): + # Don't call the close() method of the parent class, because the + # selector is mocked + self._closed = True + def _make_self_pipe(self): self._ssock = mock.Mock() self._csock = mock.Mock() @@ -40,7 +45,7 @@ self.selector = mock.Mock() self.selector.select.return_value = [] self.loop = TestBaseSelectorEventLoop(self.selector) - self.set_event_loop(self.loop, cleanup=False) + self.set_event_loop(self.loop) def test_make_socket_transport(self): m = mock.Mock() @@ -76,6 +81,15 @@ self.loop._make_ssl_transport(m, m, m, m) def test_close(self): + class EventLoop(BaseSelectorEventLoop): + def _make_self_pipe(self): + self._ssock = mock.Mock() + self._csock = mock.Mock() + self._internal_fds += 1 + + self.loop = EventLoop(self.selector) + self.set_event_loop(self.loop) + ssock = self.loop._ssock ssock.fileno.return_value = 7 csock = self.loop._csock -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:31:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:31:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjQy?= =?utf-8?q?=3A_asyncio=2ESubprocessStreamProtocol_now_closes_the_subproces?= =?utf-8?q?s?= Message-ID: <20150115122510.11561.88534@psf.io> https://hg.python.org/cpython/rev/df493e9c6821 changeset: 94165:df493e9c6821 branch: 3.4 parent: 94163:3c37825d85d3 user: Victor Stinner date: Thu Jan 15 13:16:02 2015 +0100 summary: Issue #23242: asyncio.SubprocessStreamProtocol now closes the subprocess transport at subprocess exit. Clear also its reference to the transport. files: Lib/asyncio/subprocess.py | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -94,8 +94,11 @@ reader.set_exception(exc) def process_exited(self): + returncode = self._transport.get_returncode() + self._transport.close() + self._transport = None + # wake up futures waiting for wait() - returncode = self._transport.get_returncode() while self._waiters: waiter = self._waiters.popleft() if not waiter.cancelled(): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:31:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:31:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjQz?= =?utf-8?q?=3A_Close_explicitly_transports_in_asyncio_tests?= Message-ID: <20150115122511.125890.51050@psf.io> https://hg.python.org/cpython/rev/f9b127188d43 changeset: 94169:f9b127188d43 branch: 3.4 user: Victor Stinner date: Thu Jan 15 13:18:32 2015 +0100 summary: Issue #23243: Close explicitly transports in asyncio tests files: Lib/test/test_asyncio/test_proactor_events.py | 87 +- Lib/test/test_asyncio/test_selector_events.py | 262 ++++----- Lib/test/test_asyncio/test_unix_events.py | 154 ++--- 3 files changed, 226 insertions(+), 277 deletions(-) diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -12,6 +12,15 @@ from asyncio import test_utils +def close_transport(transport): + # Don't call transport.close() because the event loop and the IOCP proactor + # are mocked + if transport._sock is None: + return + transport._sock.close() + transport._sock = None + + class ProactorSocketTransportTests(test_utils.TestCase): def setUp(self): @@ -22,17 +31,22 @@ self.protocol = test_utils.make_test_protocol(asyncio.Protocol) self.sock = mock.Mock(socket.socket) + def socket_transport(self, waiter=None): + transport = _ProactorSocketTransport(self.loop, self.sock, + self.protocol, waiter=waiter) + self.addCleanup(close_transport, transport) + return transport + def test_ctor(self): fut = asyncio.Future(loop=self.loop) - tr = _ProactorSocketTransport( - self.loop, self.sock, self.protocol, fut) + tr = self.socket_transport(waiter=fut) test_utils.run_briefly(self.loop) self.assertIsNone(fut.result()) self.protocol.connection_made(tr) self.proactor.recv.assert_called_with(self.sock, 4096) def test_loop_reading(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._loop_reading() self.loop._proactor.recv.assert_called_with(self.sock, 4096) self.assertFalse(self.protocol.data_received.called) @@ -42,8 +56,7 @@ res = asyncio.Future(loop=self.loop) res.set_result(b'data') - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - + tr = self.socket_transport() tr._read_fut = res tr._loop_reading(res) self.loop._proactor.recv.assert_called_with(self.sock, 4096) @@ -53,8 +66,7 @@ res = asyncio.Future(loop=self.loop) res.set_result(b'') - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - + tr = self.socket_transport() self.assertRaises(AssertionError, tr._loop_reading, res) tr.close = mock.Mock() @@ -67,7 +79,7 @@ def test_loop_reading_aborted(self): err = self.loop._proactor.recv.side_effect = ConnectionAbortedError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._fatal_error = mock.Mock() tr._loop_reading() tr._fatal_error.assert_called_with( @@ -77,7 +89,7 @@ def test_loop_reading_aborted_closing(self): self.loop._proactor.recv.side_effect = ConnectionAbortedError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._closing = True tr._fatal_error = mock.Mock() tr._loop_reading() @@ -85,7 +97,7 @@ def test_loop_reading_aborted_is_fatal(self): self.loop._proactor.recv.side_effect = ConnectionAbortedError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._closing = False tr._fatal_error = mock.Mock() tr._loop_reading() @@ -94,7 +106,7 @@ def test_loop_reading_conn_reset_lost(self): err = self.loop._proactor.recv.side_effect = ConnectionResetError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._closing = False tr._fatal_error = mock.Mock() tr._force_close = mock.Mock() @@ -105,7 +117,7 @@ def test_loop_reading_exception(self): err = self.loop._proactor.recv.side_effect = (OSError()) - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._fatal_error = mock.Mock() tr._loop_reading() tr._fatal_error.assert_called_with( @@ -113,19 +125,19 @@ 'Fatal read error on pipe transport') def test_write(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._loop_writing = mock.Mock() tr.write(b'data') self.assertEqual(tr._buffer, None) tr._loop_writing.assert_called_with(data=b'data') def test_write_no_data(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr.write(b'') self.assertFalse(tr._buffer) def test_write_more(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._write_fut = mock.Mock() tr._loop_writing = mock.Mock() tr.write(b'data') @@ -133,7 +145,7 @@ self.assertFalse(tr._loop_writing.called) def test_loop_writing(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._buffer = bytearray(b'data') tr._loop_writing() self.loop._proactor.send.assert_called_with(self.sock, b'data') @@ -143,7 +155,7 @@ @mock.patch('asyncio.proactor_events.logger') def test_loop_writing_err(self, m_log): err = self.loop._proactor.send.side_effect = OSError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._fatal_error = mock.Mock() tr._buffer = [b'da', b'ta'] tr._loop_writing() @@ -164,7 +176,7 @@ fut = asyncio.Future(loop=self.loop) fut.set_result(b'data') - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._write_fut = fut tr._loop_writing(fut) self.assertIsNone(tr._write_fut) @@ -173,7 +185,7 @@ fut = asyncio.Future(loop=self.loop) fut.set_result(1) - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._write_fut = fut tr.close() tr._loop_writing(fut) @@ -182,13 +194,13 @@ self.protocol.connection_lost.assert_called_with(None) def test_abort(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._force_close = mock.Mock() tr.abort() tr._force_close.assert_called_with(None) def test_close(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr.close() test_utils.run_briefly(self.loop) self.protocol.connection_lost.assert_called_with(None) @@ -201,14 +213,14 @@ self.assertFalse(self.protocol.connection_lost.called) def test_close_write_fut(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._write_fut = mock.Mock() tr.close() test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.connection_lost.called) def test_close_buffer(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._buffer = [b'data'] tr.close() test_utils.run_briefly(self.loop) @@ -216,14 +228,14 @@ @mock.patch('asyncio.base_events.logger') def test_fatal_error(self, m_logging): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._force_close = mock.Mock() tr._fatal_error(None) self.assertTrue(tr._force_close.called) self.assertTrue(m_logging.error.called) def test_force_close(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._buffer = [b'data'] read_fut = tr._read_fut = mock.Mock() write_fut = tr._write_fut = mock.Mock() @@ -237,14 +249,14 @@ self.assertEqual(tr._conn_lost, 1) def test_force_close_idempotent(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._closing = True tr._force_close(None) test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.connection_lost.called) def test_fatal_error_2(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._buffer = [b'data'] tr._force_close(None) @@ -253,14 +265,13 @@ self.assertEqual(None, tr._buffer) def test_call_connection_lost(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._call_connection_lost(None) self.assertTrue(self.protocol.connection_lost.called) self.assertTrue(self.sock.close.called) def test_write_eof(self): - tr = _ProactorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.assertTrue(tr.can_write_eof()) tr.write_eof() self.sock.shutdown.assert_called_with(socket.SHUT_WR) @@ -269,7 +280,7 @@ tr.close() def test_write_eof_buffer(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() f = asyncio.Future(loop=self.loop) tr._loop._proactor.send.return_value = f tr.write(b'data') @@ -313,11 +324,10 @@ self.assertFalse(tr.can_write_eof()) with self.assertRaises(NotImplementedError): tr.write_eof() - tr.close() + close_transport(tr) def test_pause_resume_reading(self): - tr = _ProactorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() futures = [] for msg in [b'data1', b'data2', b'data3', b'data4', b'']: f = asyncio.Future(loop=self.loop) @@ -345,10 +355,7 @@ def pause_writing_transport(self, high): - tr = _ProactorSocketTransport( - self.loop, self.sock, self.protocol) - self.addCleanup(tr.close) - + tr = self.socket_transport() tr.set_write_buffer_limits(high=high) self.assertEqual(tr.get_write_buffer_size(), 0) @@ -439,7 +446,7 @@ return (self.ssock, self.csock) self.loop = EventLoop(self.proactor) - self.set_event_loop(self.loop, cleanup=False) + self.set_event_loop(self.loop) @mock.patch.object(BaseProactorEventLoop, 'call_soon') @mock.patch.object(BaseProactorEventLoop, '_socketpair') @@ -451,6 +458,7 @@ self.assertIs(loop._csock, csock) self.assertEqual(loop._internal_fds, 1) call_soon.assert_called_with(loop._loop_self_reading) + loop.close() def test_close_self_pipe(self): self.loop._close_self_pipe() @@ -497,6 +505,7 @@ def test_make_socket_transport(self): tr = self.loop._make_socket_transport(self.sock, asyncio.Protocol()) self.assertIsInstance(tr, _ProactorSocketTransport) + close_transport(tr) def test_loop_self_reading(self): self.loop._loop_self_reading() diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -39,6 +39,15 @@ return bytearray().join(l) +def close_transport(transport): + # Don't call transport.close() because the event loop and the selector + # are mocked + if transport._sock is None: + return + transport._sock.close() + transport._sock = None + + class BaseSelectorEventLoopTests(test_utils.TestCase): def setUp(self): @@ -52,6 +61,7 @@ self.loop.add_reader = mock.Mock() transport = self.loop._make_socket_transport(m, asyncio.Protocol()) self.assertIsInstance(transport, _SelectorSocketTransport) + close_transport(transport) @unittest.skipIf(ssl is None, 'No ssl module') def test_make_ssl_transport(self): @@ -64,11 +74,19 @@ with test_utils.disable_logger(): transport = self.loop._make_ssl_transport( m, asyncio.Protocol(), m, waiter) + # execute the handshake while the logger is disabled + # to ignore SSL handshake failure + test_utils.run_briefly(self.loop) + # Sanity check class_name = transport.__class__.__name__ self.assertIn("ssl", class_name.lower()) self.assertIn("transport", class_name.lower()) + transport.close() + # execute pending callbacks to close the socket transport + test_utils.run_briefly(self.loop) + @mock.patch('asyncio.selector_events.ssl', None) @mock.patch('asyncio.sslproto.ssl', None) def test_make_ssl_transport_without_ssl_error(self): @@ -650,21 +668,27 @@ self.sock = mock.Mock(socket.socket) self.sock.fileno.return_value = 7 + def create_transport(self): + transport = _SelectorTransport(self.loop, self.sock, self.protocol, + None) + self.addCleanup(close_transport, transport) + return transport + def test_ctor(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() self.assertIs(tr._loop, self.loop) self.assertIs(tr._sock, self.sock) self.assertIs(tr._sock_fd, 7) def test_abort(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr._force_close = mock.Mock() tr.abort() tr._force_close.assert_called_with(None) def test_close(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr.close() self.assertTrue(tr._closing) @@ -677,7 +701,7 @@ self.assertEqual(1, self.loop.remove_reader_count[7]) def test_close_write_buffer(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr._buffer.extend(b'data') tr.close() @@ -686,7 +710,7 @@ self.assertFalse(self.protocol.connection_lost.called) def test_force_close(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr._buffer.extend(b'1') self.loop.add_reader(7, mock.sentinel) self.loop.add_writer(7, mock.sentinel) @@ -705,7 +729,7 @@ @mock.patch('asyncio.log.logger.error') def test_fatal_error(self, m_exc): exc = OSError() - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr._force_close = mock.Mock() tr._fatal_error(exc) @@ -718,7 +742,7 @@ def test_connection_lost(self): exc = OSError() - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) tr._call_connection_lost(exc) @@ -739,9 +763,14 @@ self.sock = mock.Mock(socket.socket) self.sock_fd = self.sock.fileno.return_value = 7 + def socket_transport(self, waiter=None): + transport = _SelectorSocketTransport(self.loop, self.sock, + self.protocol, waiter=waiter) + self.addCleanup(close_transport, transport) + return transport + def test_ctor(self): - tr = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.loop.assert_reader(7, tr._read_ready) test_utils.run_briefly(self.loop) self.protocol.connection_made.assert_called_with(tr) @@ -749,14 +778,12 @@ def test_ctor_with_waiter(self): fut = asyncio.Future(loop=self.loop) - _SelectorSocketTransport( - self.loop, self.sock, self.protocol, fut) + self.socket_transport(waiter=fut) test_utils.run_briefly(self.loop) self.assertIsNone(fut.result()) def test_pause_resume_reading(self): - tr = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.assertFalse(tr._paused) self.loop.assert_reader(7, tr._read_ready) tr.pause_reading() @@ -769,8 +796,7 @@ tr.resume_reading() def test_read_ready(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() self.sock.recv.return_value = b'data' transport._read_ready() @@ -778,8 +804,7 @@ self.protocol.data_received.assert_called_with(b'data') def test_read_ready_eof(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.close = mock.Mock() self.sock.recv.return_value = b'' @@ -789,8 +814,7 @@ transport.close.assert_called_with() def test_read_ready_eof_keep_open(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.close = mock.Mock() self.sock.recv.return_value = b'' @@ -804,8 +828,7 @@ def test_read_ready_tryagain(self, m_exc): self.sock.recv.side_effect = BlockingIOError - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport._read_ready() @@ -815,8 +838,7 @@ def test_read_ready_tryagain_interrupted(self, m_exc): self.sock.recv.side_effect = InterruptedError - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport._read_ready() @@ -826,8 +848,7 @@ def test_read_ready_conn_reset(self, m_exc): err = self.sock.recv.side_effect = ConnectionResetError() - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._force_close = mock.Mock() with test_utils.disable_logger(): transport._read_ready() @@ -837,8 +858,7 @@ def test_read_ready_err(self, m_exc): err = self.sock.recv.side_effect = OSError() - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport._read_ready() @@ -850,8 +870,7 @@ data = b'data' self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.sock.send.assert_called_with(data) @@ -859,8 +878,7 @@ data = bytearray(b'data') self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.sock.send.assert_called_with(data) self.assertEqual(data, bytearray(b'data')) # Hasn't been mutated. @@ -869,22 +887,19 @@ data = memoryview(b'data') self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.sock.send.assert_called_with(data) def test_write_no_data(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(b'data') transport.write(b'') self.assertFalse(self.sock.send.called) self.assertEqual(list_to_buffer([b'data']), transport._buffer) def test_write_buffer(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(b'data1') transport.write(b'data2') self.assertFalse(self.sock.send.called) @@ -895,8 +910,7 @@ data = b'data' self.sock.send.return_value = 2 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -906,8 +920,7 @@ data = bytearray(b'data') self.sock.send.return_value = 2 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -918,8 +931,7 @@ data = memoryview(b'data') self.sock.send.return_value = 2 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -930,8 +942,7 @@ self.sock.send.return_value = 0 self.sock.fileno.return_value = 7 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -941,8 +952,7 @@ self.sock.send.side_effect = BlockingIOError data = b'data' - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -953,8 +963,7 @@ err = self.sock.send.side_effect = OSError() data = b'data' - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport.write(data) transport._fatal_error.assert_called_with( @@ -973,13 +982,11 @@ m_log.warning.assert_called_with('socket.send() raised exception.') def test_write_str(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() self.assertRaises(TypeError, transport.write, 'str') def test_write_closing(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.close() self.assertEqual(transport._conn_lost, 1) transport.write(b'data') @@ -989,8 +996,7 @@ data = b'data' self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(data) self.loop.add_writer(7, transport._write_ready) transport._write_ready() @@ -1001,8 +1007,7 @@ data = b'data' self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._closing = True transport._buffer.extend(data) self.loop.add_writer(7, transport._write_ready) @@ -1013,8 +1018,7 @@ self.protocol.connection_lost.assert_called_with(None) def test_write_ready_no_data(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() # This is an internal error. self.assertRaises(AssertionError, transport._write_ready) @@ -1022,8 +1026,7 @@ data = b'data' self.sock.send.return_value = 2 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(data) self.loop.add_writer(7, transport._write_ready) transport._write_ready() @@ -1034,8 +1037,7 @@ data = b'data' self.sock.send.return_value = 0 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(data) self.loop.add_writer(7, transport._write_ready) transport._write_ready() @@ -1045,8 +1047,7 @@ def test_write_ready_tryagain(self): self.sock.send.side_effect = BlockingIOError - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer = list_to_buffer([b'data1', b'data2']) self.loop.add_writer(7, transport._write_ready) transport._write_ready() @@ -1057,8 +1058,7 @@ def test_write_ready_exception(self): err = self.sock.send.side_effect = OSError() - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport._buffer.extend(b'data') transport._write_ready() @@ -1071,16 +1071,14 @@ self.sock.send.side_effect = OSError() remove_writer = self.loop.remove_writer = mock.Mock() - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.close() transport._buffer.extend(b'data') transport._write_ready() remove_writer.assert_called_with(self.sock_fd) def test_write_eof(self): - tr = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.assertTrue(tr.can_write_eof()) tr.write_eof() self.sock.shutdown.assert_called_with(socket.SHUT_WR) @@ -1089,8 +1087,7 @@ tr.close() def test_write_eof_buffer(self): - tr = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.sock.send.side_effect = BlockingIOError tr.write(b'data') tr.write_eof() @@ -1117,9 +1114,15 @@ self.sslcontext = mock.Mock() self.sslcontext.wrap_socket.return_value = self.sslsock + def ssl_transport(self, waiter=None, server_hostname=None): + transport = _SelectorSslTransport(self.loop, self.sock, self.protocol, + self.sslcontext, waiter=waiter, + server_hostname=server_hostname) + self.addCleanup(close_transport, transport) + return transport + def _make_one(self, create_waiter=None): - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext) + transport = self.ssl_transport() self.sock.reset_mock() self.sslsock.reset_mock() self.sslcontext.reset_mock() @@ -1128,9 +1131,7 @@ def test_on_handshake(self): waiter = asyncio.Future(loop=self.loop) - tr = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, - waiter=waiter) + tr = self.ssl_transport(waiter=waiter) self.assertTrue(self.sslsock.do_handshake.called) self.loop.assert_reader(1, tr._read_ready) test_utils.run_briefly(self.loop) @@ -1139,15 +1140,13 @@ def test_on_handshake_reader_retry(self): self.loop.set_debug(False) self.sslsock.do_handshake.side_effect = ssl.SSLWantReadError - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext) + transport = self.ssl_transport() self.loop.assert_reader(1, transport._on_handshake, None) def test_on_handshake_writer_retry(self): self.loop.set_debug(False) self.sslsock.do_handshake.side_effect = ssl.SSLWantWriteError - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext) + transport = self.ssl_transport() self.loop.assert_writer(1, transport._on_handshake, None) def test_on_handshake_exc(self): @@ -1155,16 +1154,14 @@ self.sslsock.do_handshake.side_effect = exc with test_utils.disable_logger(): waiter = asyncio.Future(loop=self.loop) - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, waiter) + transport = self.ssl_transport(waiter=waiter) self.assertTrue(waiter.done()) self.assertIs(exc, waiter.exception()) self.assertTrue(self.sslsock.close.called) def test_on_handshake_base_exc(self): waiter = asyncio.Future(loop=self.loop) - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, waiter) + transport = self.ssl_transport(waiter=waiter) exc = BaseException() self.sslsock.do_handshake.side_effect = exc with test_utils.disable_logger(): @@ -1177,8 +1174,7 @@ # Python issue #23197: cancelling an handshake must not raise an # exception or log an error, even if the handshake failed waiter = asyncio.Future(loop=self.loop) - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, waiter) + transport = self.ssl_transport(waiter=waiter) waiter.cancel() exc = ValueError() self.sslsock.do_handshake.side_effect = exc @@ -1437,9 +1433,7 @@ @unittest.skipIf(ssl is None, 'No SSL support') def test_server_hostname(self): - _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, - server_hostname='localhost') + self.ssl_transport(server_hostname='localhost') self.sslcontext.wrap_socket.assert_called_with( self.sock, do_handshake_on_connect=False, server_side=False, server_hostname='localhost') @@ -1462,9 +1456,15 @@ self.sock = mock.Mock(spec_set=socket.socket) self.sock.fileno.return_value = 7 + def datagram_transport(self, address=None): + transport = _SelectorDatagramTransport(self.loop, self.sock, + self.protocol, + address=address) + self.addCleanup(close_transport, transport) + return transport + def test_read_ready(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() self.sock.recvfrom.return_value = (b'data', ('0.0.0.0', 1234)) transport._read_ready() @@ -1473,8 +1473,7 @@ b'data', ('0.0.0.0', 1234)) def test_read_ready_tryagain(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() self.sock.recvfrom.side_effect = BlockingIOError transport._fatal_error = mock.Mock() @@ -1483,8 +1482,7 @@ self.assertFalse(transport._fatal_error.called) def test_read_ready_err(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() err = self.sock.recvfrom.side_effect = RuntimeError() transport._fatal_error = mock.Mock() @@ -1495,8 +1493,7 @@ 'Fatal read error on datagram transport') def test_read_ready_oserr(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() err = self.sock.recvfrom.side_effect = OSError() transport._fatal_error = mock.Mock() @@ -1507,8 +1504,7 @@ def test_sendto(self): data = b'data' - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport.sendto(data, ('0.0.0.0', 1234)) self.assertTrue(self.sock.sendto.called) self.assertEqual( @@ -1516,8 +1512,7 @@ def test_sendto_bytearray(self): data = bytearray(b'data') - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport.sendto(data, ('0.0.0.0', 1234)) self.assertTrue(self.sock.sendto.called) self.assertEqual( @@ -1525,16 +1520,14 @@ def test_sendto_memoryview(self): data = memoryview(b'data') - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport.sendto(data, ('0.0.0.0', 1234)) self.assertTrue(self.sock.sendto.called) self.assertEqual( self.sock.sendto.call_args[0], (data, ('0.0.0.0', 1234))) def test_sendto_no_data(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((b'data', ('0.0.0.0', 12345))) transport.sendto(b'', ()) self.assertFalse(self.sock.sendto.called) @@ -1542,8 +1535,7 @@ [(b'data', ('0.0.0.0', 12345))], list(transport._buffer)) def test_sendto_buffer(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((b'data1', ('0.0.0.0', 12345))) transport.sendto(b'data2', ('0.0.0.0', 12345)) self.assertFalse(self.sock.sendto.called) @@ -1554,8 +1546,7 @@ def test_sendto_buffer_bytearray(self): data2 = bytearray(b'data2') - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((b'data1', ('0.0.0.0', 12345))) transport.sendto(data2, ('0.0.0.0', 12345)) self.assertFalse(self.sock.sendto.called) @@ -1567,8 +1558,7 @@ def test_sendto_buffer_memoryview(self): data2 = memoryview(b'data2') - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((b'data1', ('0.0.0.0', 12345))) transport.sendto(data2, ('0.0.0.0', 12345)) self.assertFalse(self.sock.sendto.called) @@ -1583,8 +1573,7 @@ self.sock.sendto.side_effect = BlockingIOError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport.sendto(data, ('0.0.0.0', 12345)) self.loop.assert_writer(7, transport._sendto_ready) @@ -1596,8 +1585,7 @@ data = b'data' err = self.sock.sendto.side_effect = RuntimeError() - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._fatal_error = mock.Mock() transport.sendto(data, ()) @@ -1620,8 +1608,7 @@ self.sock.sendto.side_effect = ConnectionRefusedError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._fatal_error = mock.Mock() transport.sendto(data, ()) @@ -1633,8 +1620,7 @@ self.sock.send.side_effect = ConnectionRefusedError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) + transport = self.datagram_transport(address=('0.0.0.0', 1)) transport._fatal_error = mock.Mock() transport.sendto(data) @@ -1642,19 +1628,16 @@ self.assertTrue(self.protocol.error_received.called) def test_sendto_str(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() self.assertRaises(TypeError, transport.sendto, 'str', ()) def test_sendto_connected_addr(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) + transport = self.datagram_transport(address=('0.0.0.0', 1)) self.assertRaises( ValueError, transport.sendto, b'str', ('0.0.0.0', 2)) def test_sendto_closing(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, address=(1,)) + transport = self.datagram_transport(address=(1,)) transport.close() self.assertEqual(transport._conn_lost, 1) transport.sendto(b'data', (1,)) @@ -1664,8 +1647,7 @@ data = b'data' self.sock.sendto.return_value = len(data) - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((data, ('0.0.0.0', 12345))) self.loop.add_writer(7, transport._sendto_ready) transport._sendto_ready() @@ -1678,8 +1660,7 @@ data = b'data' self.sock.send.return_value = len(data) - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._closing = True transport._buffer.append((data, ())) self.loop.add_writer(7, transport._sendto_ready) @@ -1690,8 +1671,7 @@ self.protocol.connection_lost.assert_called_with(None) def test_sendto_ready_no_data(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() self.loop.add_writer(7, transport._sendto_ready) transport._sendto_ready() self.assertFalse(self.sock.sendto.called) @@ -1700,8 +1680,7 @@ def test_sendto_ready_tryagain(self): self.sock.sendto.side_effect = BlockingIOError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.extend([(b'data1', ()), (b'data2', ())]) self.loop.add_writer(7, transport._sendto_ready) transport._sendto_ready() @@ -1714,8 +1693,7 @@ def test_sendto_ready_exception(self): err = self.sock.sendto.side_effect = RuntimeError() - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() @@ -1727,8 +1705,7 @@ def test_sendto_ready_error_received(self): self.sock.sendto.side_effect = ConnectionRefusedError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() @@ -1738,8 +1715,7 @@ def test_sendto_ready_error_received_connection(self): self.sock.send.side_effect = ConnectionRefusedError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) + transport = self.datagram_transport(address=('0.0.0.0', 1)) transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() @@ -1749,8 +1725,7 @@ @mock.patch('asyncio.base_events.logger.error') def test_fatal_error_connected(self, m_exc): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) + transport = self.datagram_transport(address=('0.0.0.0', 1)) err = ConnectionRefusedError() transport._fatal_error(err) self.assertFalse(self.protocol.error_received.called) @@ -1758,7 +1733,6 @@ test_utils.MockPattern( 'Fatal error on transport\nprotocol:.*\ntransport:.*'), exc_info=(ConnectionRefusedError, MOCK_ANY, MOCK_ANY)) - transport.close() if __name__ == '__main__': diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -26,6 +26,15 @@ MOCK_ANY = mock.ANY +def close_pipe_transport(transport): + # Don't call transport.close() because the event loop and the selector + # are mocked + if transport._pipe is None: + return + transport._pipe.close() + transport._pipe = None + + @unittest.skipUnless(signal, 'Signals are not supported') class SelectorEventLoopSignalTests(test_utils.TestCase): @@ -333,24 +342,28 @@ m_fstat.return_value = st self.addCleanup(fstat_patcher.stop) + def read_pipe_transport(self, waiter=None): + transport = unix_events._UnixReadPipeTransport(self.loop, self.pipe, + self.protocol, + waiter=waiter) + self.addCleanup(close_pipe_transport, transport) + return transport + def test_ctor(self): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() self.loop.assert_reader(5, tr._read_ready) test_utils.run_briefly(self.loop) self.protocol.connection_made.assert_called_with(tr) def test_ctor_with_waiter(self): fut = asyncio.Future(loop=self.loop) - unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol, fut) + tr = self.read_pipe_transport(waiter=fut) test_utils.run_briefly(self.loop) self.assertIsNone(fut.result()) @mock.patch('os.read') def test__read_ready(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() m_read.return_value = b'data' tr._read_ready() @@ -359,8 +372,7 @@ @mock.patch('os.read') def test__read_ready_eof(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() m_read.return_value = b'' tr._read_ready() @@ -372,8 +384,7 @@ @mock.patch('os.read') def test__read_ready_blocked(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() m_read.side_effect = BlockingIOError tr._read_ready() @@ -384,8 +395,7 @@ @mock.patch('asyncio.log.logger.error') @mock.patch('os.read') def test__read_ready_error(self, m_read, m_logexc): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() err = OSError() m_read.side_effect = err tr._close = mock.Mock() @@ -401,9 +411,7 @@ @mock.patch('os.read') def test_pause_reading(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() m = mock.Mock() self.loop.add_reader(5, m) tr.pause_reading() @@ -411,26 +419,20 @@ @mock.patch('os.read') def test_resume_reading(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() tr.resume_reading() self.loop.assert_reader(5, tr._read_ready) @mock.patch('os.read') def test_close(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() tr._close = mock.Mock() tr.close() tr._close.assert_called_with(None) @mock.patch('os.read') def test_close_already_closing(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() tr._closing = True tr._close = mock.Mock() tr.close() @@ -438,9 +440,7 @@ @mock.patch('os.read') def test__close(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() err = object() tr._close(err) self.assertTrue(tr._closing) @@ -449,8 +449,7 @@ self.protocol.connection_lost.assert_called_with(err) def test__call_connection_lost(self): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) @@ -463,8 +462,7 @@ self.assertIsNone(tr._loop) def test__call_connection_lost_with_err(self): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) @@ -496,31 +494,33 @@ m_fstat.return_value = st self.addCleanup(fstat_patcher.stop) + def write_pipe_transport(self, waiter=None): + transport = unix_events._UnixWritePipeTransport(self.loop, self.pipe, + self.protocol, + waiter=waiter) + self.addCleanup(close_pipe_transport, transport) + return transport + def test_ctor(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.loop.assert_reader(5, tr._read_ready) test_utils.run_briefly(self.loop) self.protocol.connection_made.assert_called_with(tr) def test_ctor_with_waiter(self): fut = asyncio.Future(loop=self.loop) - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol, fut) + tr = self.write_pipe_transport(waiter=fut) self.loop.assert_reader(5, tr._read_ready) test_utils.run_briefly(self.loop) self.assertEqual(None, fut.result()) def test_can_write_eof(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.assertTrue(tr.can_write_eof()) @mock.patch('os.write') def test_write(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() m_write.return_value = 4 tr.write(b'data') m_write.assert_called_with(5, b'data') @@ -529,9 +529,7 @@ @mock.patch('os.write') def test_write_no_data(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() tr.write(b'') self.assertFalse(m_write.called) self.assertFalse(self.loop.writers) @@ -539,9 +537,7 @@ @mock.patch('os.write') def test_write_partial(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() m_write.return_value = 2 tr.write(b'data') m_write.assert_called_with(5, b'data') @@ -550,9 +546,7 @@ @mock.patch('os.write') def test_write_buffer(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'previous'] tr.write(b'data') @@ -562,9 +556,7 @@ @mock.patch('os.write') def test_write_again(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() m_write.side_effect = BlockingIOError() tr.write(b'data') m_write.assert_called_with(5, b'data') @@ -574,9 +566,7 @@ @mock.patch('asyncio.unix_events.logger') @mock.patch('os.write') def test_write_err(self, m_write, m_log): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() err = OSError() m_write.side_effect = err tr._fatal_error = mock.Mock() @@ -602,8 +592,7 @@ @mock.patch('os.write') def test_write_close(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() tr._read_ready() # pipe was closed by peer tr.write(b'data') @@ -612,8 +601,7 @@ self.assertEqual(tr._conn_lost, 2) def test__read_ready(self): - tr = unix_events._UnixWritePipeTransport(self.loop, self.pipe, - self.protocol) + tr = self.write_pipe_transport() tr._read_ready() self.assertFalse(self.loop.readers) self.assertFalse(self.loop.writers) @@ -623,8 +611,7 @@ @mock.patch('os.write') def test__write_ready(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.return_value = 4 @@ -635,9 +622,7 @@ @mock.patch('os.write') def test__write_ready_partial(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.return_value = 3 @@ -648,9 +633,7 @@ @mock.patch('os.write') def test__write_ready_again(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.side_effect = BlockingIOError() @@ -661,9 +644,7 @@ @mock.patch('os.write') def test__write_ready_empty(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.return_value = 0 @@ -675,9 +656,7 @@ @mock.patch('asyncio.log.logger.error') @mock.patch('os.write') def test__write_ready_err(self, m_write, m_logexc): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.side_effect = err = OSError() @@ -698,9 +677,7 @@ @mock.patch('os.write') def test__write_ready_closing(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._closing = True tr._buffer = [b'da', b'ta'] @@ -715,9 +692,7 @@ @mock.patch('os.write') def test_abort(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) self.loop.add_reader(5, tr._read_ready) tr._buffer = [b'da', b'ta'] @@ -731,8 +706,7 @@ self.protocol.connection_lost.assert_called_with(None) def test__call_connection_lost(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) @@ -745,8 +719,7 @@ self.assertIsNone(tr._loop) def test__call_connection_lost_with_err(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) @@ -759,9 +732,7 @@ self.assertIsNone(tr._loop) def test_close(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() tr.write_eof = mock.Mock() tr.close() tr.write_eof.assert_called_with() @@ -770,18 +741,14 @@ tr.close() def test_close_closing(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() tr.write_eof = mock.Mock() tr._closing = True tr.close() self.assertFalse(tr.write_eof.called) def test_write_eof(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() tr.write_eof() self.assertTrue(tr._closing) self.assertFalse(self.loop.readers) @@ -789,8 +756,7 @@ self.protocol.connection_lost.assert_called_with(None) def test_write_eof_pending(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() tr._buffer = [b'data'] tr.write_eof() self.assertTrue(tr._closing) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:31:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:31:20 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150115122511.11628.34373@psf.io> https://hg.python.org/cpython/rev/dc21d72d0f87 changeset: 94170:dc21d72d0f87 parent: 94164:69285bf7e865 parent: 94169:f9b127188d43 user: Victor Stinner date: Thu Jan 15 13:23:36 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/sslproto.py | 1 + Lib/asyncio/subprocess.py | 5 +- Lib/asyncio/unix_events.py | 2 +- Lib/test/test_asyncio/test_base_events.py | 1 + Lib/test/test_asyncio/test_proactor_events.py | 91 +- Lib/test/test_asyncio/test_selector_events.py | 278 ++++----- Lib/test/test_asyncio/test_unix_events.py | 157 ++--- 7 files changed, 255 insertions(+), 280 deletions(-) diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -417,6 +417,7 @@ self._session_established = False self._in_handshake = False self._in_shutdown = False + self._transport = None def connection_made(self, transport): """Called when the low-level connection is made. diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -94,8 +94,11 @@ reader.set_exception(exc) def process_exited(self): + returncode = self._transport.get_returncode() + self._transport.close() + self._transport = None + # wake up futures waiting for wait() - returncode = self._transport.get_returncode() while self._waiters: waiter = self._waiters.popleft() if not waiter.cancelled(): diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -516,7 +516,7 @@ self._loop.call_soon(self._call_connection_lost, None) def close(self): - if not self._closing: + if self._pipe is not None and not self._closing: # write_eof is all what we needed to close the write pipe self.write_eof() diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -590,6 +590,7 @@ raise ValueError('spam') loop = Loop() + self.addCleanup(loop.close) asyncio.set_event_loop(loop) def run_loop(): diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -12,26 +12,41 @@ from asyncio import test_utils +def close_transport(transport): + # Don't call transport.close() because the event loop and the IOCP proactor + # are mocked + if transport._sock is None: + return + transport._sock.close() + transport._sock = None + + class ProactorSocketTransportTests(test_utils.TestCase): def setUp(self): self.loop = self.new_test_loop() + self.addCleanup(self.loop.close) self.proactor = mock.Mock() self.loop._proactor = self.proactor self.protocol = test_utils.make_test_protocol(asyncio.Protocol) self.sock = mock.Mock(socket.socket) + def socket_transport(self, waiter=None): + transport = _ProactorSocketTransport(self.loop, self.sock, + self.protocol, waiter=waiter) + self.addCleanup(close_transport, transport) + return transport + def test_ctor(self): fut = asyncio.Future(loop=self.loop) - tr = _ProactorSocketTransport( - self.loop, self.sock, self.protocol, fut) + tr = self.socket_transport(waiter=fut) test_utils.run_briefly(self.loop) self.assertIsNone(fut.result()) self.protocol.connection_made(tr) self.proactor.recv.assert_called_with(self.sock, 4096) def test_loop_reading(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._loop_reading() self.loop._proactor.recv.assert_called_with(self.sock, 4096) self.assertFalse(self.protocol.data_received.called) @@ -41,8 +56,7 @@ res = asyncio.Future(loop=self.loop) res.set_result(b'data') - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - + tr = self.socket_transport() tr._read_fut = res tr._loop_reading(res) self.loop._proactor.recv.assert_called_with(self.sock, 4096) @@ -52,8 +66,7 @@ res = asyncio.Future(loop=self.loop) res.set_result(b'') - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - + tr = self.socket_transport() self.assertRaises(AssertionError, tr._loop_reading, res) tr.close = mock.Mock() @@ -66,7 +79,7 @@ def test_loop_reading_aborted(self): err = self.loop._proactor.recv.side_effect = ConnectionAbortedError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._fatal_error = mock.Mock() tr._loop_reading() tr._fatal_error.assert_called_with( @@ -76,7 +89,7 @@ def test_loop_reading_aborted_closing(self): self.loop._proactor.recv.side_effect = ConnectionAbortedError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._closing = True tr._fatal_error = mock.Mock() tr._loop_reading() @@ -84,7 +97,7 @@ def test_loop_reading_aborted_is_fatal(self): self.loop._proactor.recv.side_effect = ConnectionAbortedError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._closing = False tr._fatal_error = mock.Mock() tr._loop_reading() @@ -93,7 +106,7 @@ def test_loop_reading_conn_reset_lost(self): err = self.loop._proactor.recv.side_effect = ConnectionResetError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._closing = False tr._fatal_error = mock.Mock() tr._force_close = mock.Mock() @@ -104,7 +117,7 @@ def test_loop_reading_exception(self): err = self.loop._proactor.recv.side_effect = (OSError()) - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._fatal_error = mock.Mock() tr._loop_reading() tr._fatal_error.assert_called_with( @@ -112,19 +125,19 @@ 'Fatal read error on pipe transport') def test_write(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._loop_writing = mock.Mock() tr.write(b'data') self.assertEqual(tr._buffer, None) tr._loop_writing.assert_called_with(data=b'data') def test_write_no_data(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr.write(b'') self.assertFalse(tr._buffer) def test_write_more(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._write_fut = mock.Mock() tr._loop_writing = mock.Mock() tr.write(b'data') @@ -132,7 +145,7 @@ self.assertFalse(tr._loop_writing.called) def test_loop_writing(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._buffer = bytearray(b'data') tr._loop_writing() self.loop._proactor.send.assert_called_with(self.sock, b'data') @@ -142,7 +155,7 @@ @mock.patch('asyncio.proactor_events.logger') def test_loop_writing_err(self, m_log): err = self.loop._proactor.send.side_effect = OSError() - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._fatal_error = mock.Mock() tr._buffer = [b'da', b'ta'] tr._loop_writing() @@ -163,7 +176,7 @@ fut = asyncio.Future(loop=self.loop) fut.set_result(b'data') - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._write_fut = fut tr._loop_writing(fut) self.assertIsNone(tr._write_fut) @@ -172,7 +185,7 @@ fut = asyncio.Future(loop=self.loop) fut.set_result(1) - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._write_fut = fut tr.close() tr._loop_writing(fut) @@ -181,13 +194,13 @@ self.protocol.connection_lost.assert_called_with(None) def test_abort(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._force_close = mock.Mock() tr.abort() tr._force_close.assert_called_with(None) def test_close(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr.close() test_utils.run_briefly(self.loop) self.protocol.connection_lost.assert_called_with(None) @@ -200,14 +213,14 @@ self.assertFalse(self.protocol.connection_lost.called) def test_close_write_fut(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._write_fut = mock.Mock() tr.close() test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.connection_lost.called) def test_close_buffer(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._buffer = [b'data'] tr.close() test_utils.run_briefly(self.loop) @@ -215,14 +228,14 @@ @mock.patch('asyncio.base_events.logger') def test_fatal_error(self, m_logging): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._force_close = mock.Mock() tr._fatal_error(None) self.assertTrue(tr._force_close.called) self.assertTrue(m_logging.error.called) def test_force_close(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._buffer = [b'data'] read_fut = tr._read_fut = mock.Mock() write_fut = tr._write_fut = mock.Mock() @@ -236,14 +249,14 @@ self.assertEqual(tr._conn_lost, 1) def test_force_close_idempotent(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._closing = True tr._force_close(None) test_utils.run_briefly(self.loop) self.assertFalse(self.protocol.connection_lost.called) def test_fatal_error_2(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._buffer = [b'data'] tr._force_close(None) @@ -252,14 +265,13 @@ self.assertEqual(None, tr._buffer) def test_call_connection_lost(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() tr._call_connection_lost(None) self.assertTrue(self.protocol.connection_lost.called) self.assertTrue(self.sock.close.called) def test_write_eof(self): - tr = _ProactorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.assertTrue(tr.can_write_eof()) tr.write_eof() self.sock.shutdown.assert_called_with(socket.SHUT_WR) @@ -268,7 +280,7 @@ tr.close() def test_write_eof_buffer(self): - tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) + tr = self.socket_transport() f = asyncio.Future(loop=self.loop) tr._loop._proactor.send.return_value = f tr.write(b'data') @@ -312,11 +324,10 @@ self.assertFalse(tr.can_write_eof()) with self.assertRaises(NotImplementedError): tr.write_eof() - tr.close() + close_transport(tr) def test_pause_resume_reading(self): - tr = _ProactorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() futures = [] for msg in [b'data1', b'data2', b'data3', b'data4', b'']: f = asyncio.Future(loop=self.loop) @@ -344,10 +355,7 @@ def pause_writing_transport(self, high): - tr = _ProactorSocketTransport( - self.loop, self.sock, self.protocol) - self.addCleanup(tr.close) - + tr = self.socket_transport() tr.set_write_buffer_limits(high=high) self.assertEqual(tr.get_write_buffer_size(), 0) @@ -438,7 +446,7 @@ return (self.ssock, self.csock) self.loop = EventLoop(self.proactor) - self.set_event_loop(self.loop, cleanup=False) + self.set_event_loop(self.loop) @mock.patch.object(BaseProactorEventLoop, 'call_soon') @mock.patch.object(BaseProactorEventLoop, '_socketpair') @@ -450,6 +458,7 @@ self.assertIs(loop._csock, csock) self.assertEqual(loop._internal_fds, 1) call_soon.assert_called_with(loop._loop_self_reading) + loop.close() def test_close_self_pipe(self): self.loop._close_self_pipe() @@ -459,6 +468,9 @@ self.assertIsNone(self.loop._ssock) self.assertIsNone(self.loop._csock) + # Don't call close(): _close_self_pipe() cannot be called twice + self.loop._closed = True + def test_close(self): self.loop._close_self_pipe = mock.Mock() self.loop.close() @@ -493,6 +505,7 @@ def test_make_socket_transport(self): tr = self.loop._make_socket_transport(self.sock, asyncio.Protocol()) self.assertIsInstance(tr, _ProactorSocketTransport) + close_transport(tr) def test_loop_self_reading(self): self.loop._loop_self_reading() diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -24,6 +24,11 @@ class TestBaseSelectorEventLoop(BaseSelectorEventLoop): + def close(self): + # Don't call the close() method of the parent class, because the + # selector is mocked + self._closed = True + def _make_self_pipe(self): self._ssock = mock.Mock() self._csock = mock.Mock() @@ -34,19 +39,29 @@ return bytearray().join(l) +def close_transport(transport): + # Don't call transport.close() because the event loop and the selector + # are mocked + if transport._sock is None: + return + transport._sock.close() + transport._sock = None + + class BaseSelectorEventLoopTests(test_utils.TestCase): def setUp(self): self.selector = mock.Mock() self.selector.select.return_value = [] self.loop = TestBaseSelectorEventLoop(self.selector) - self.set_event_loop(self.loop, cleanup=False) + self.set_event_loop(self.loop) def test_make_socket_transport(self): m = mock.Mock() self.loop.add_reader = mock.Mock() transport = self.loop._make_socket_transport(m, asyncio.Protocol()) self.assertIsInstance(transport, _SelectorSocketTransport) + close_transport(transport) @unittest.skipIf(ssl is None, 'No ssl module') def test_make_ssl_transport(self): @@ -59,11 +74,19 @@ with test_utils.disable_logger(): transport = self.loop._make_ssl_transport( m, asyncio.Protocol(), m, waiter) + # execute the handshake while the logger is disabled + # to ignore SSL handshake failure + test_utils.run_briefly(self.loop) + # Sanity check class_name = transport.__class__.__name__ self.assertIn("ssl", class_name.lower()) self.assertIn("transport", class_name.lower()) + transport.close() + # execute pending callbacks to close the socket transport + test_utils.run_briefly(self.loop) + @mock.patch('asyncio.selector_events.ssl', None) @mock.patch('asyncio.sslproto.ssl', None) def test_make_ssl_transport_without_ssl_error(self): @@ -76,6 +99,15 @@ self.loop._make_ssl_transport(m, m, m, m) def test_close(self): + class EventLoop(BaseSelectorEventLoop): + def _make_self_pipe(self): + self._ssock = mock.Mock() + self._csock = mock.Mock() + self._internal_fds += 1 + + self.loop = EventLoop(self.selector) + self.set_event_loop(self.loop) + ssock = self.loop._ssock ssock.fileno.return_value = 7 csock = self.loop._csock @@ -636,21 +668,27 @@ self.sock = mock.Mock(socket.socket) self.sock.fileno.return_value = 7 + def create_transport(self): + transport = _SelectorTransport(self.loop, self.sock, self.protocol, + None) + self.addCleanup(close_transport, transport) + return transport + def test_ctor(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() self.assertIs(tr._loop, self.loop) self.assertIs(tr._sock, self.sock) self.assertIs(tr._sock_fd, 7) def test_abort(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr._force_close = mock.Mock() tr.abort() tr._force_close.assert_called_with(None) def test_close(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr.close() self.assertTrue(tr._closing) @@ -663,7 +701,7 @@ self.assertEqual(1, self.loop.remove_reader_count[7]) def test_close_write_buffer(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr._buffer.extend(b'data') tr.close() @@ -672,7 +710,7 @@ self.assertFalse(self.protocol.connection_lost.called) def test_force_close(self): - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr._buffer.extend(b'1') self.loop.add_reader(7, mock.sentinel) self.loop.add_writer(7, mock.sentinel) @@ -691,7 +729,7 @@ @mock.patch('asyncio.log.logger.error') def test_fatal_error(self, m_exc): exc = OSError() - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() tr._force_close = mock.Mock() tr._fatal_error(exc) @@ -704,7 +742,7 @@ def test_connection_lost(self): exc = OSError() - tr = _SelectorTransport(self.loop, self.sock, self.protocol, None) + tr = self.create_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) tr._call_connection_lost(exc) @@ -725,9 +763,14 @@ self.sock = mock.Mock(socket.socket) self.sock_fd = self.sock.fileno.return_value = 7 + def socket_transport(self, waiter=None): + transport = _SelectorSocketTransport(self.loop, self.sock, + self.protocol, waiter=waiter) + self.addCleanup(close_transport, transport) + return transport + def test_ctor(self): - tr = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.loop.assert_reader(7, tr._read_ready) test_utils.run_briefly(self.loop) self.protocol.connection_made.assert_called_with(tr) @@ -735,14 +778,12 @@ def test_ctor_with_waiter(self): fut = asyncio.Future(loop=self.loop) - _SelectorSocketTransport( - self.loop, self.sock, self.protocol, fut) + self.socket_transport(waiter=fut) test_utils.run_briefly(self.loop) self.assertIsNone(fut.result()) def test_pause_resume_reading(self): - tr = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.assertFalse(tr._paused) self.loop.assert_reader(7, tr._read_ready) tr.pause_reading() @@ -755,8 +796,7 @@ tr.resume_reading() def test_read_ready(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() self.sock.recv.return_value = b'data' transport._read_ready() @@ -764,8 +804,7 @@ self.protocol.data_received.assert_called_with(b'data') def test_read_ready_eof(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.close = mock.Mock() self.sock.recv.return_value = b'' @@ -775,8 +814,7 @@ transport.close.assert_called_with() def test_read_ready_eof_keep_open(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.close = mock.Mock() self.sock.recv.return_value = b'' @@ -790,8 +828,7 @@ def test_read_ready_tryagain(self, m_exc): self.sock.recv.side_effect = BlockingIOError - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport._read_ready() @@ -801,8 +838,7 @@ def test_read_ready_tryagain_interrupted(self, m_exc): self.sock.recv.side_effect = InterruptedError - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport._read_ready() @@ -812,8 +848,7 @@ def test_read_ready_conn_reset(self, m_exc): err = self.sock.recv.side_effect = ConnectionResetError() - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._force_close = mock.Mock() with test_utils.disable_logger(): transport._read_ready() @@ -823,8 +858,7 @@ def test_read_ready_err(self, m_exc): err = self.sock.recv.side_effect = OSError() - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport._read_ready() @@ -836,8 +870,7 @@ data = b'data' self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.sock.send.assert_called_with(data) @@ -845,8 +878,7 @@ data = bytearray(b'data') self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.sock.send.assert_called_with(data) self.assertEqual(data, bytearray(b'data')) # Hasn't been mutated. @@ -855,22 +887,19 @@ data = memoryview(b'data') self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.sock.send.assert_called_with(data) def test_write_no_data(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(b'data') transport.write(b'') self.assertFalse(self.sock.send.called) self.assertEqual(list_to_buffer([b'data']), transport._buffer) def test_write_buffer(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(b'data1') transport.write(b'data2') self.assertFalse(self.sock.send.called) @@ -881,8 +910,7 @@ data = b'data' self.sock.send.return_value = 2 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -892,8 +920,7 @@ data = bytearray(b'data') self.sock.send.return_value = 2 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -904,8 +931,7 @@ data = memoryview(b'data') self.sock.send.return_value = 2 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -916,8 +942,7 @@ self.sock.send.return_value = 0 self.sock.fileno.return_value = 7 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -927,8 +952,7 @@ self.sock.send.side_effect = BlockingIOError data = b'data' - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.write(data) self.loop.assert_writer(7, transport._write_ready) @@ -939,8 +963,7 @@ err = self.sock.send.side_effect = OSError() data = b'data' - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport.write(data) transport._fatal_error.assert_called_with( @@ -959,13 +982,11 @@ m_log.warning.assert_called_with('socket.send() raised exception.') def test_write_str(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() self.assertRaises(TypeError, transport.write, 'str') def test_write_closing(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.close() self.assertEqual(transport._conn_lost, 1) transport.write(b'data') @@ -975,8 +996,7 @@ data = b'data' self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(data) self.loop.add_writer(7, transport._write_ready) transport._write_ready() @@ -987,8 +1007,7 @@ data = b'data' self.sock.send.return_value = len(data) - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._closing = True transport._buffer.extend(data) self.loop.add_writer(7, transport._write_ready) @@ -999,8 +1018,7 @@ self.protocol.connection_lost.assert_called_with(None) def test_write_ready_no_data(self): - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() # This is an internal error. self.assertRaises(AssertionError, transport._write_ready) @@ -1008,8 +1026,7 @@ data = b'data' self.sock.send.return_value = 2 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(data) self.loop.add_writer(7, transport._write_ready) transport._write_ready() @@ -1020,8 +1037,7 @@ data = b'data' self.sock.send.return_value = 0 - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer.extend(data) self.loop.add_writer(7, transport._write_ready) transport._write_ready() @@ -1031,8 +1047,7 @@ def test_write_ready_tryagain(self): self.sock.send.side_effect = BlockingIOError - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._buffer = list_to_buffer([b'data1', b'data2']) self.loop.add_writer(7, transport._write_ready) transport._write_ready() @@ -1043,8 +1058,7 @@ def test_write_ready_exception(self): err = self.sock.send.side_effect = OSError() - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport._fatal_error = mock.Mock() transport._buffer.extend(b'data') transport._write_ready() @@ -1057,16 +1071,14 @@ self.sock.send.side_effect = OSError() remove_writer = self.loop.remove_writer = mock.Mock() - transport = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + transport = self.socket_transport() transport.close() transport._buffer.extend(b'data') transport._write_ready() remove_writer.assert_called_with(self.sock_fd) def test_write_eof(self): - tr = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.assertTrue(tr.can_write_eof()) tr.write_eof() self.sock.shutdown.assert_called_with(socket.SHUT_WR) @@ -1075,8 +1087,7 @@ tr.close() def test_write_eof_buffer(self): - tr = _SelectorSocketTransport( - self.loop, self.sock, self.protocol) + tr = self.socket_transport() self.sock.send.side_effect = BlockingIOError tr.write(b'data') tr.write_eof() @@ -1103,9 +1114,15 @@ self.sslcontext = mock.Mock() self.sslcontext.wrap_socket.return_value = self.sslsock + def ssl_transport(self, waiter=None, server_hostname=None): + transport = _SelectorSslTransport(self.loop, self.sock, self.protocol, + self.sslcontext, waiter=waiter, + server_hostname=server_hostname) + self.addCleanup(close_transport, transport) + return transport + def _make_one(self, create_waiter=None): - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext) + transport = self.ssl_transport() self.sock.reset_mock() self.sslsock.reset_mock() self.sslcontext.reset_mock() @@ -1114,9 +1131,7 @@ def test_on_handshake(self): waiter = asyncio.Future(loop=self.loop) - tr = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, - waiter=waiter) + tr = self.ssl_transport(waiter=waiter) self.assertTrue(self.sslsock.do_handshake.called) self.loop.assert_reader(1, tr._read_ready) test_utils.run_briefly(self.loop) @@ -1125,15 +1140,13 @@ def test_on_handshake_reader_retry(self): self.loop.set_debug(False) self.sslsock.do_handshake.side_effect = ssl.SSLWantReadError - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext) + transport = self.ssl_transport() self.loop.assert_reader(1, transport._on_handshake, None) def test_on_handshake_writer_retry(self): self.loop.set_debug(False) self.sslsock.do_handshake.side_effect = ssl.SSLWantWriteError - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext) + transport = self.ssl_transport() self.loop.assert_writer(1, transport._on_handshake, None) def test_on_handshake_exc(self): @@ -1141,16 +1154,14 @@ self.sslsock.do_handshake.side_effect = exc with test_utils.disable_logger(): waiter = asyncio.Future(loop=self.loop) - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, waiter) + transport = self.ssl_transport(waiter=waiter) self.assertTrue(waiter.done()) self.assertIs(exc, waiter.exception()) self.assertTrue(self.sslsock.close.called) def test_on_handshake_base_exc(self): waiter = asyncio.Future(loop=self.loop) - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, waiter) + transport = self.ssl_transport(waiter=waiter) exc = BaseException() self.sslsock.do_handshake.side_effect = exc with test_utils.disable_logger(): @@ -1163,8 +1174,7 @@ # Python issue #23197: cancelling an handshake must not raise an # exception or log an error, even if the handshake failed waiter = asyncio.Future(loop=self.loop) - transport = _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, waiter) + transport = self.ssl_transport(waiter=waiter) waiter.cancel() exc = ValueError() self.sslsock.do_handshake.side_effect = exc @@ -1423,9 +1433,7 @@ @unittest.skipIf(ssl is None, 'No SSL support') def test_server_hostname(self): - _SelectorSslTransport( - self.loop, self.sock, self.protocol, self.sslcontext, - server_hostname='localhost') + self.ssl_transport(server_hostname='localhost') self.sslcontext.wrap_socket.assert_called_with( self.sock, do_handshake_on_connect=False, server_side=False, server_hostname='localhost') @@ -1448,9 +1456,15 @@ self.sock = mock.Mock(spec_set=socket.socket) self.sock.fileno.return_value = 7 + def datagram_transport(self, address=None): + transport = _SelectorDatagramTransport(self.loop, self.sock, + self.protocol, + address=address) + self.addCleanup(close_transport, transport) + return transport + def test_read_ready(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() self.sock.recvfrom.return_value = (b'data', ('0.0.0.0', 1234)) transport._read_ready() @@ -1459,8 +1473,7 @@ b'data', ('0.0.0.0', 1234)) def test_read_ready_tryagain(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() self.sock.recvfrom.side_effect = BlockingIOError transport._fatal_error = mock.Mock() @@ -1469,8 +1482,7 @@ self.assertFalse(transport._fatal_error.called) def test_read_ready_err(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() err = self.sock.recvfrom.side_effect = RuntimeError() transport._fatal_error = mock.Mock() @@ -1481,8 +1493,7 @@ 'Fatal read error on datagram transport') def test_read_ready_oserr(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() err = self.sock.recvfrom.side_effect = OSError() transport._fatal_error = mock.Mock() @@ -1493,8 +1504,7 @@ def test_sendto(self): data = b'data' - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport.sendto(data, ('0.0.0.0', 1234)) self.assertTrue(self.sock.sendto.called) self.assertEqual( @@ -1502,8 +1512,7 @@ def test_sendto_bytearray(self): data = bytearray(b'data') - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport.sendto(data, ('0.0.0.0', 1234)) self.assertTrue(self.sock.sendto.called) self.assertEqual( @@ -1511,16 +1520,14 @@ def test_sendto_memoryview(self): data = memoryview(b'data') - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport.sendto(data, ('0.0.0.0', 1234)) self.assertTrue(self.sock.sendto.called) self.assertEqual( self.sock.sendto.call_args[0], (data, ('0.0.0.0', 1234))) def test_sendto_no_data(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((b'data', ('0.0.0.0', 12345))) transport.sendto(b'', ()) self.assertFalse(self.sock.sendto.called) @@ -1528,8 +1535,7 @@ [(b'data', ('0.0.0.0', 12345))], list(transport._buffer)) def test_sendto_buffer(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((b'data1', ('0.0.0.0', 12345))) transport.sendto(b'data2', ('0.0.0.0', 12345)) self.assertFalse(self.sock.sendto.called) @@ -1540,8 +1546,7 @@ def test_sendto_buffer_bytearray(self): data2 = bytearray(b'data2') - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((b'data1', ('0.0.0.0', 12345))) transport.sendto(data2, ('0.0.0.0', 12345)) self.assertFalse(self.sock.sendto.called) @@ -1553,8 +1558,7 @@ def test_sendto_buffer_memoryview(self): data2 = memoryview(b'data2') - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((b'data1', ('0.0.0.0', 12345))) transport.sendto(data2, ('0.0.0.0', 12345)) self.assertFalse(self.sock.sendto.called) @@ -1569,8 +1573,7 @@ self.sock.sendto.side_effect = BlockingIOError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport.sendto(data, ('0.0.0.0', 12345)) self.loop.assert_writer(7, transport._sendto_ready) @@ -1582,8 +1585,7 @@ data = b'data' err = self.sock.sendto.side_effect = RuntimeError() - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._fatal_error = mock.Mock() transport.sendto(data, ()) @@ -1606,8 +1608,7 @@ self.sock.sendto.side_effect = ConnectionRefusedError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._fatal_error = mock.Mock() transport.sendto(data, ()) @@ -1619,8 +1620,7 @@ self.sock.send.side_effect = ConnectionRefusedError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) + transport = self.datagram_transport(address=('0.0.0.0', 1)) transport._fatal_error = mock.Mock() transport.sendto(data) @@ -1628,19 +1628,16 @@ self.assertTrue(self.protocol.error_received.called) def test_sendto_str(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() self.assertRaises(TypeError, transport.sendto, 'str', ()) def test_sendto_connected_addr(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) + transport = self.datagram_transport(address=('0.0.0.0', 1)) self.assertRaises( ValueError, transport.sendto, b'str', ('0.0.0.0', 2)) def test_sendto_closing(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, address=(1,)) + transport = self.datagram_transport(address=(1,)) transport.close() self.assertEqual(transport._conn_lost, 1) transport.sendto(b'data', (1,)) @@ -1650,8 +1647,7 @@ data = b'data' self.sock.sendto.return_value = len(data) - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.append((data, ('0.0.0.0', 12345))) self.loop.add_writer(7, transport._sendto_ready) transport._sendto_ready() @@ -1664,8 +1660,7 @@ data = b'data' self.sock.send.return_value = len(data) - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._closing = True transport._buffer.append((data, ())) self.loop.add_writer(7, transport._sendto_ready) @@ -1676,8 +1671,7 @@ self.protocol.connection_lost.assert_called_with(None) def test_sendto_ready_no_data(self): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() self.loop.add_writer(7, transport._sendto_ready) transport._sendto_ready() self.assertFalse(self.sock.sendto.called) @@ -1686,8 +1680,7 @@ def test_sendto_ready_tryagain(self): self.sock.sendto.side_effect = BlockingIOError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._buffer.extend([(b'data1', ()), (b'data2', ())]) self.loop.add_writer(7, transport._sendto_ready) transport._sendto_ready() @@ -1700,8 +1693,7 @@ def test_sendto_ready_exception(self): err = self.sock.sendto.side_effect = RuntimeError() - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() @@ -1713,8 +1705,7 @@ def test_sendto_ready_error_received(self): self.sock.sendto.side_effect = ConnectionRefusedError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol) + transport = self.datagram_transport() transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() @@ -1724,8 +1715,7 @@ def test_sendto_ready_error_received_connection(self): self.sock.send.side_effect = ConnectionRefusedError - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) + transport = self.datagram_transport(address=('0.0.0.0', 1)) transport._fatal_error = mock.Mock() transport._buffer.append((b'data', ())) transport._sendto_ready() @@ -1735,8 +1725,7 @@ @mock.patch('asyncio.base_events.logger.error') def test_fatal_error_connected(self, m_exc): - transport = _SelectorDatagramTransport( - self.loop, self.sock, self.protocol, ('0.0.0.0', 1)) + transport = self.datagram_transport(address=('0.0.0.0', 1)) err = ConnectionRefusedError() transport._fatal_error(err) self.assertFalse(self.protocol.error_received.called) @@ -1744,7 +1733,6 @@ test_utils.MockPattern( 'Fatal error on transport\nprotocol:.*\ntransport:.*'), exc_info=(ConnectionRefusedError, MOCK_ANY, MOCK_ANY)) - transport.close() if __name__ == '__main__': diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -26,6 +26,15 @@ MOCK_ANY = mock.ANY +def close_pipe_transport(transport): + # Don't call transport.close() because the event loop and the selector + # are mocked + if transport._pipe is None: + return + transport._pipe.close() + transport._pipe = None + + @unittest.skipUnless(signal, 'Signals are not supported') class SelectorEventLoopSignalTests(test_utils.TestCase): @@ -333,24 +342,28 @@ m_fstat.return_value = st self.addCleanup(fstat_patcher.stop) + def read_pipe_transport(self, waiter=None): + transport = unix_events._UnixReadPipeTransport(self.loop, self.pipe, + self.protocol, + waiter=waiter) + self.addCleanup(close_pipe_transport, transport) + return transport + def test_ctor(self): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() self.loop.assert_reader(5, tr._read_ready) test_utils.run_briefly(self.loop) self.protocol.connection_made.assert_called_with(tr) def test_ctor_with_waiter(self): fut = asyncio.Future(loop=self.loop) - unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol, fut) + tr = self.read_pipe_transport(waiter=fut) test_utils.run_briefly(self.loop) self.assertIsNone(fut.result()) @mock.patch('os.read') def test__read_ready(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() m_read.return_value = b'data' tr._read_ready() @@ -359,8 +372,7 @@ @mock.patch('os.read') def test__read_ready_eof(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() m_read.return_value = b'' tr._read_ready() @@ -372,8 +384,7 @@ @mock.patch('os.read') def test__read_ready_blocked(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() m_read.side_effect = BlockingIOError tr._read_ready() @@ -384,8 +395,7 @@ @mock.patch('asyncio.log.logger.error') @mock.patch('os.read') def test__read_ready_error(self, m_read, m_logexc): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() err = OSError() m_read.side_effect = err tr._close = mock.Mock() @@ -401,9 +411,7 @@ @mock.patch('os.read') def test_pause_reading(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() m = mock.Mock() self.loop.add_reader(5, m) tr.pause_reading() @@ -411,26 +419,20 @@ @mock.patch('os.read') def test_resume_reading(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() tr.resume_reading() self.loop.assert_reader(5, tr._read_ready) @mock.patch('os.read') def test_close(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() tr._close = mock.Mock() tr.close() tr._close.assert_called_with(None) @mock.patch('os.read') def test_close_already_closing(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() tr._closing = True tr._close = mock.Mock() tr.close() @@ -438,9 +440,7 @@ @mock.patch('os.read') def test__close(self, m_read): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.read_pipe_transport() err = object() tr._close(err) self.assertTrue(tr._closing) @@ -449,8 +449,7 @@ self.protocol.connection_lost.assert_called_with(err) def test__call_connection_lost(self): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) @@ -463,8 +462,7 @@ self.assertIsNone(tr._loop) def test__call_connection_lost_with_err(self): - tr = unix_events._UnixReadPipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.read_pipe_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) @@ -496,31 +494,33 @@ m_fstat.return_value = st self.addCleanup(fstat_patcher.stop) + def write_pipe_transport(self, waiter=None): + transport = unix_events._UnixWritePipeTransport(self.loop, self.pipe, + self.protocol, + waiter=waiter) + self.addCleanup(close_pipe_transport, transport) + return transport + def test_ctor(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.loop.assert_reader(5, tr._read_ready) test_utils.run_briefly(self.loop) self.protocol.connection_made.assert_called_with(tr) def test_ctor_with_waiter(self): fut = asyncio.Future(loop=self.loop) - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol, fut) + tr = self.write_pipe_transport(waiter=fut) self.loop.assert_reader(5, tr._read_ready) test_utils.run_briefly(self.loop) self.assertEqual(None, fut.result()) def test_can_write_eof(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.assertTrue(tr.can_write_eof()) @mock.patch('os.write') def test_write(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() m_write.return_value = 4 tr.write(b'data') m_write.assert_called_with(5, b'data') @@ -529,9 +529,7 @@ @mock.patch('os.write') def test_write_no_data(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() tr.write(b'') self.assertFalse(m_write.called) self.assertFalse(self.loop.writers) @@ -539,9 +537,7 @@ @mock.patch('os.write') def test_write_partial(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() m_write.return_value = 2 tr.write(b'data') m_write.assert_called_with(5, b'data') @@ -550,9 +546,7 @@ @mock.patch('os.write') def test_write_buffer(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'previous'] tr.write(b'data') @@ -562,9 +556,7 @@ @mock.patch('os.write') def test_write_again(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() m_write.side_effect = BlockingIOError() tr.write(b'data') m_write.assert_called_with(5, b'data') @@ -574,9 +566,7 @@ @mock.patch('asyncio.unix_events.logger') @mock.patch('os.write') def test_write_err(self, m_write, m_log): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() err = OSError() m_write.side_effect = err tr._fatal_error = mock.Mock() @@ -602,8 +592,7 @@ @mock.patch('os.write') def test_write_close(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() tr._read_ready() # pipe was closed by peer tr.write(b'data') @@ -612,8 +601,7 @@ self.assertEqual(tr._conn_lost, 2) def test__read_ready(self): - tr = unix_events._UnixWritePipeTransport(self.loop, self.pipe, - self.protocol) + tr = self.write_pipe_transport() tr._read_ready() self.assertFalse(self.loop.readers) self.assertFalse(self.loop.writers) @@ -623,8 +611,7 @@ @mock.patch('os.write') def test__write_ready(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.return_value = 4 @@ -635,9 +622,7 @@ @mock.patch('os.write') def test__write_ready_partial(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.return_value = 3 @@ -648,9 +633,7 @@ @mock.patch('os.write') def test__write_ready_again(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.side_effect = BlockingIOError() @@ -661,9 +644,7 @@ @mock.patch('os.write') def test__write_ready_empty(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.return_value = 0 @@ -675,9 +656,7 @@ @mock.patch('asyncio.log.logger.error') @mock.patch('os.write') def test__write_ready_err(self, m_write, m_logexc): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._buffer = [b'da', b'ta'] m_write.side_effect = err = OSError() @@ -698,9 +677,7 @@ @mock.patch('os.write') def test__write_ready_closing(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) tr._closing = True tr._buffer = [b'da', b'ta'] @@ -715,9 +692,7 @@ @mock.patch('os.write') def test_abort(self, m_write): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() self.loop.add_writer(5, tr._write_ready) self.loop.add_reader(5, tr._read_ready) tr._buffer = [b'da', b'ta'] @@ -731,8 +706,7 @@ self.protocol.connection_lost.assert_called_with(None) def test__call_connection_lost(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) @@ -745,8 +719,7 @@ self.assertIsNone(tr._loop) def test__call_connection_lost_with_err(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() self.assertIsNotNone(tr._protocol) self.assertIsNotNone(tr._loop) @@ -759,26 +732,23 @@ self.assertIsNone(tr._loop) def test_close(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() tr.write_eof = mock.Mock() tr.close() tr.write_eof.assert_called_with() + # closing the transport twice must not fail + tr.close() + def test_close_closing(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() tr.write_eof = mock.Mock() tr._closing = True tr.close() self.assertFalse(tr.write_eof.called) def test_write_eof(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) - + tr = self.write_pipe_transport() tr.write_eof() self.assertTrue(tr._closing) self.assertFalse(self.loop.readers) @@ -786,8 +756,7 @@ self.protocol.connection_lost.assert_called_with(None) def test_write_eof_pending(self): - tr = unix_events._UnixWritePipeTransport( - self.loop, self.pipe, self.protocol) + tr = self.write_pipe_transport() tr._buffer = [b'data'] tr.write_eof() self.assertTrue(tr._closing) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:42:38 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:42:38 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?b?IF9Qcm9hY3RvckJhc2VQaXBlVHJhbnNwb3J0Ll9fcmVwcl9fKCk=?= Message-ID: <20150115124236.11579.95428@psf.io> https://hg.python.org/cpython/rev/063a34c7bf77 changeset: 94171:063a34c7bf77 branch: 3.4 parent: 94169:f9b127188d43 user: Victor Stinner date: Thu Jan 15 13:32:28 2015 +0100 summary: asyncio: Fix _ProactorBasePipeTransport.__repr__() Check if the _sock attribute is None to check if the transport is closed. files: Lib/asyncio/proactor_events.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -43,12 +43,12 @@ def __repr__(self): info = [self.__class__.__name__] - fd = self._sock.fileno() - if fd < 0: + if self._sock is None: info.append('closed') elif self._closing: info.append('closing') - info.append('fd=%s' % fd) + if self._sock is not None: + info.append('fd=%s' % self._sock.fileno()) if self._read_fut is not None: info.append('read=%s' % self._read_fut) if self._write_fut is not None: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:42:38 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:42:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150115124236.22423.35150@psf.io> https://hg.python.org/cpython/rev/831d71f2811d changeset: 94173:831d71f2811d parent: 94170:dc21d72d0f87 parent: 94172:a35b790a2db3 user: Victor Stinner date: Thu Jan 15 13:41:01 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/proactor_events.py | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -43,12 +43,12 @@ def __repr__(self): info = [self.__class__.__name__] - fd = self._sock.fileno() - if fd < 0: + if self._sock is None: info.append('closed') elif self._closing: info.append('closing') - info.append('fd=%s' % fd) + if self._sock is not None: + info.append('fd=%s' % self._sock.fileno()) if self._read_fut is not None: info.append('read=%s' % self._read_fut) if self._write_fut is not None: @@ -72,6 +72,7 @@ self._loop.call_soon(self._call_connection_lost, None) if self._read_fut is not None: self._read_fut.cancel() + self._read_fut = None def _fatal_error(self, exc, message='Fatal error on pipe transport'): if isinstance(exc, (BrokenPipeError, ConnectionResetError)): @@ -93,9 +94,10 @@ self._conn_lost += 1 if self._write_fut: self._write_fut.cancel() + self._write_fut = None if self._read_fut: self._read_fut.cancel() - self._write_fut = self._read_fut = None + self._read_fut = None self._pending_write = 0 self._buffer = None self._loop.call_soon(self._call_connection_lost, exc) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 13:42:38 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 12:42:38 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?q?_=5FProactorBasePipeTransport=2Eclose=28=29?= Message-ID: <20150115124236.22421.3089@psf.io> https://hg.python.org/cpython/rev/a35b790a2db3 changeset: 94172:a35b790a2db3 branch: 3.4 user: Victor Stinner date: Thu Jan 15 13:40:27 2015 +0100 summary: asyncio: Fix _ProactorBasePipeTransport.close() Set the _read_fut attribute to None after cancelling it. This change should fix a race condition with _ProactorWritePipeTransport._pipe_closed(). files: Lib/asyncio/proactor_events.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -72,6 +72,7 @@ self._loop.call_soon(self._call_connection_lost, None) if self._read_fut is not None: self._read_fut.cancel() + self._read_fut = None def _fatal_error(self, exc, message='Fatal error on pipe transport'): if isinstance(exc, (BrokenPipeError, ConnectionResetError)): @@ -93,9 +94,10 @@ self._conn_lost += 1 if self._write_fut: self._write_fut.cancel() + self._write_fut = None if self._read_fut: self._read_fut.cancel() - self._write_fut = self._read_fut = None + self._read_fut = None self._pending_write = 0 self._buffer = None self._loop.call_soon(self._call_connection_lost, exc) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 14:26:29 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 13:26:29 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQ2xv?= =?utf-8?q?se_the_transport_on_subprocess_creation_failure?= Message-ID: <20150115132626.125904.51748@psf.io> https://hg.python.org/cpython/rev/6bace35f55e3 changeset: 94174:6bace35f55e3 branch: 3.4 parent: 94172:a35b790a2db3 user: Victor Stinner date: Thu Jan 15 14:24:22 2015 +0100 summary: asyncio: Close the transport on subprocess creation failure files: Lib/asyncio/unix_events.py | 6 +++++- Lib/asyncio/windows_events.py | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -177,7 +177,11 @@ transp = _UnixSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra=extra, **kwargs) - yield from transp._post_init() + try: + yield from transp._post_init() + except: + transp.close() + raise watcher.add_child_handler(transp.get_pid(), self._child_watcher_callback, transp) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -272,7 +272,12 @@ transp = _WindowsSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra=extra, **kwargs) - yield from transp._post_init() + try: + yield from transp._post_init() + except: + transp.close() + raise + return transp -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 14:26:29 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 13:26:29 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQ2xv?= =?utf-8?q?se_transports_in_tests?= Message-ID: <20150115132626.22407.40933@psf.io> https://hg.python.org/cpython/rev/9dfd33f3657f changeset: 94175:9dfd33f3657f branch: 3.4 user: Victor Stinner date: Thu Jan 15 14:24:55 2015 +0100 summary: asyncio: Close transports in tests * Use test_utils.run_briefly() to execute pending calls to really close transports * sslproto: mock also _SSLPipe.shutdown(), it's need to close the transport * pipe test: the test doesn't close explicitly the PipeHandle, so ignore the warning instead * test_popen: use the context manager ("with p:") to explicitly close pipes files: Lib/test/test_asyncio/test_selector_events.py | 2 + Lib/test/test_asyncio/test_sslproto.py | 4 +++ Lib/test/test_asyncio/test_subprocess.py | 1 + Lib/test/test_asyncio/test_windows_utils.py | 11 +++++++-- 4 files changed, 15 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1180,6 +1180,8 @@ self.sslsock.do_handshake.side_effect = exc with test_utils.disable_logger(): transport._on_handshake(0) + transport.close() + test_utils.run_briefly(self.loop) def test_pause_resume_reading(self): tr = self._make_one() diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -33,6 +33,7 @@ waiter.cancel() transport = mock.Mock() sslpipe = mock.Mock() + sslpipe.shutdown.return_value = b'' sslpipe.do_handshake.side_effect = do_handshake with mock.patch('asyncio.sslproto._SSLPipe', return_value=sslpipe): ssl_proto.connection_made(transport) @@ -40,6 +41,9 @@ with test_utils.disable_logger(): self.loop.run_until_complete(handshake_fut) + # Close the transport + ssl_proto._app_transport.close() + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -286,6 +286,7 @@ # "Exception during subprocess creation, kill the subprocess" with test_utils.disable_logger(): self.loop.run_until_complete(cancel_make_transport()) + test_utils.run_briefly(self.loop) if sys.platform != 'win32': diff --git a/Lib/test/test_asyncio/test_windows_utils.py b/Lib/test/test_asyncio/test_windows_utils.py --- a/Lib/test/test_asyncio/test_windows_utils.py +++ b/Lib/test/test_asyncio/test_windows_utils.py @@ -3,6 +3,7 @@ import socket import sys import unittest +import warnings from unittest import mock if sys.platform != 'win32': @@ -115,8 +116,10 @@ self.assertEqual(p.handle, h) # check garbage collection of p closes handle - del p - support.gc_collect() + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", "", ResourceWarning) + del p + support.gc_collect() try: _winapi.CloseHandle(h) except OSError as e: @@ -170,7 +173,9 @@ self.assertTrue(msg.upper().rstrip().startswith(out)) self.assertTrue(b"stderr".startswith(err)) - p.wait() + # The context manager calls wait() and closes resources + with p: + pass if __name__ == '__main__': -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 14:26:29 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 13:26:29 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150115132626.125888.6031@psf.io> https://hg.python.org/cpython/rev/78fb5c4e3129 changeset: 94176:78fb5c4e3129 parent: 94173:831d71f2811d parent: 94175:9dfd33f3657f user: Victor Stinner date: Thu Jan 15 14:25:08 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/unix_events.py | 6 ++++- Lib/asyncio/windows_events.py | 7 +++++- Lib/test/test_asyncio/test_selector_events.py | 2 + Lib/test/test_asyncio/test_sslproto.py | 4 +++ Lib/test/test_asyncio/test_subprocess.py | 1 + Lib/test/test_asyncio/test_windows_utils.py | 11 +++++++-- 6 files changed, 26 insertions(+), 5 deletions(-) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -177,7 +177,11 @@ transp = _UnixSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra=extra, **kwargs) - yield from transp._post_init() + try: + yield from transp._post_init() + except: + transp.close() + raise watcher.add_child_handler(transp.get_pid(), self._child_watcher_callback, transp) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -272,7 +272,12 @@ transp = _WindowsSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra=extra, **kwargs) - yield from transp._post_init() + try: + yield from transp._post_init() + except: + transp.close() + raise + return transp diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1180,6 +1180,8 @@ self.sslsock.do_handshake.side_effect = exc with test_utils.disable_logger(): transport._on_handshake(0) + transport.close() + test_utils.run_briefly(self.loop) def test_pause_resume_reading(self): tr = self._make_one() diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -33,6 +33,7 @@ waiter.cancel() transport = mock.Mock() sslpipe = mock.Mock() + sslpipe.shutdown.return_value = b'' sslpipe.do_handshake.side_effect = do_handshake with mock.patch('asyncio.sslproto._SSLPipe', return_value=sslpipe): ssl_proto.connection_made(transport) @@ -40,6 +41,9 @@ with test_utils.disable_logger(): self.loop.run_until_complete(handshake_fut) + # Close the transport + ssl_proto._app_transport.close() + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -286,6 +286,7 @@ # "Exception during subprocess creation, kill the subprocess" with test_utils.disable_logger(): self.loop.run_until_complete(cancel_make_transport()) + test_utils.run_briefly(self.loop) if sys.platform != 'win32': diff --git a/Lib/test/test_asyncio/test_windows_utils.py b/Lib/test/test_asyncio/test_windows_utils.py --- a/Lib/test/test_asyncio/test_windows_utils.py +++ b/Lib/test/test_asyncio/test_windows_utils.py @@ -3,6 +3,7 @@ import socket import sys import unittest +import warnings from unittest import mock if sys.platform != 'win32': @@ -115,8 +116,10 @@ self.assertEqual(p.handle, h) # check garbage collection of p closes handle - del p - support.gc_collect() + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", "", ResourceWarning) + del p + support.gc_collect() try: _winapi.CloseHandle(h) except OSError as e: @@ -170,7 +173,9 @@ self.assertTrue(msg.upper().rstrip().startswith(out)) self.assertTrue(b"stderr".startswith(err)) - p.wait() + # The context manager calls wait() and closes resources + with p: + pass if __name__ == '__main__': -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 16:31:09 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 15:31:09 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150115153026.125906.56311@psf.io> https://hg.python.org/cpython/rev/e05646ea3c40 changeset: 94178:e05646ea3c40 parent: 94176:78fb5c4e3129 parent: 94177:8adf1896712d user: Victor Stinner date: Thu Jan 15 16:29:23 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/tasks.py | 12 +++++-- Lib/test/test_asyncio/test_tasks.py | 27 +++++++++++++++++ 2 files changed, 35 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -347,10 +347,9 @@ it cancels the task and raises TimeoutError. To avoid the task cancellation, wrap it in shield(). - Usage: + If the wait is cancelled, the task is also cancelled. - result = yield from asyncio.wait_for(fut, 10.0) - + This function is a coroutine. """ if loop is None: loop = events.get_event_loop() @@ -367,7 +366,12 @@ try: # wait until the future completes or the timeout - yield from waiter + try: + yield from waiter + except futures.CancelledError: + fut.remove_done_callback(cb) + fut.cancel() + raise if fut.done(): return fut.result() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1705,6 +1705,33 @@ 'test_task_source_traceback')) self.loop.run_until_complete(task) + def _test_cancel_wait_for(self, timeout): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + @asyncio.coroutine + def blocking_coroutine(): + fut = asyncio.Future(loop=loop) + # Block: fut result is never set + yield from fut + + task = loop.create_task(blocking_coroutine()) + + wait = loop.create_task(asyncio.wait_for(task, timeout, loop=loop)) + loop.call_soon(wait.cancel) + + self.assertRaises(asyncio.CancelledError, + loop.run_until_complete, wait) + + # Python issue #23219: cancelling the wait must also cancel the task + self.assertTrue(task.cancelled()) + + def test_cancel_blocking_wait_for(self): + self._test_cancel_wait_for(None) + + def test_cancel_wait_for(self): + self._test_cancel_wait_for(60.0) + class GatherTestsBase: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 16:31:09 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 15:31:09 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMzIx?= =?utf-8?q?9=3A_cancelling_asyncio=2Ewait=5Ffor=28=29_now_cancels_the_task?= Message-ID: <20150115153026.72565.22664@psf.io> https://hg.python.org/cpython/rev/8adf1896712d changeset: 94177:8adf1896712d branch: 3.4 parent: 94175:9dfd33f3657f user: Victor Stinner date: Thu Jan 15 16:29:10 2015 +0100 summary: Closes #23219: cancelling asyncio.wait_for() now cancels the task files: Lib/asyncio/tasks.py | 12 +++++-- Lib/test/test_asyncio/test_tasks.py | 27 +++++++++++++++++ 2 files changed, 35 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -347,10 +347,9 @@ it cancels the task and raises TimeoutError. To avoid the task cancellation, wrap it in shield(). - Usage: + If the wait is cancelled, the task is also cancelled. - result = yield from asyncio.wait_for(fut, 10.0) - + This function is a coroutine. """ if loop is None: loop = events.get_event_loop() @@ -367,7 +366,12 @@ try: # wait until the future completes or the timeout - yield from waiter + try: + yield from waiter + except futures.CancelledError: + fut.remove_done_callback(cb) + fut.cancel() + raise if fut.done(): return fut.result() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1705,6 +1705,33 @@ 'test_task_source_traceback')) self.loop.run_until_complete(task) + def _test_cancel_wait_for(self, timeout): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + @asyncio.coroutine + def blocking_coroutine(): + fut = asyncio.Future(loop=loop) + # Block: fut result is never set + yield from fut + + task = loop.create_task(blocking_coroutine()) + + wait = loop.create_task(asyncio.wait_for(task, timeout, loop=loop)) + loop.call_soon(wait.cancel) + + self.assertRaises(asyncio.CancelledError, + loop.run_until_complete, wait) + + # Python issue #23219: cancelling the wait must also cancel the task + self.assertTrue(task.cancelled()) + + def test_cancel_blocking_wait_for(self): + self._test_cancel_wait_for(None) + + def test_cancel_wait_for(self): + self._test_cancel_wait_for(60.0) + class GatherTestsBase: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 18:12:45 2015 From: python-checkins at python.org (steve.dower) Date: Thu, 15 Jan 2015 17:12:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fixes_sys=2Ewinver_generat?= =?utf-8?q?ion_and_removes_dependency_on_user32=2Edll?= Message-ID: <20150115171218.3960.25809@psf.io> https://hg.python.org/cpython/rev/36b0a5c1bc9e changeset: 94179:36b0a5c1bc9e user: Steve Dower date: Thu Jan 15 09:10:16 2015 -0800 summary: Fixes sys.winver generation and removes dependency on user32.dll files: PC/dl_nt.c | 8 ++++++++ PCbuild/pyproject.props | 2 +- PCbuild/python.props | 8 ++++++-- PCbuild/pythoncore.vcxproj | 8 +++++--- 4 files changed, 20 insertions(+), 6 deletions(-) diff --git a/PC/dl_nt.c b/PC/dl_nt.c --- a/PC/dl_nt.c +++ b/PC/dl_nt.c @@ -12,7 +12,12 @@ #include "windows.h" #ifdef Py_ENABLE_SHARED +#ifdef MS_DLL_ID +// The string is available at build, so fill the buffer immediately +char dllVersionBuffer[16] = MS_DLL_ID; +#else char dllVersionBuffer[16] = ""; // a private buffer +#endif // Python Globals HMODULE PyWin_DLLhModule = NULL; @@ -88,8 +93,11 @@ { case DLL_PROCESS_ATTACH: PyWin_DLLhModule = hInst; +#ifndef MS_DLL_ID + // If we have MS_DLL_ID, we don't need to load the string. // 1000 is a magic number I picked out of the air. Could do with a #define, I spose... LoadString(hInst, 1000, dllVersionBuffer, sizeof(dllVersionBuffer)); +#endif #if HAVE_SXS // and capture our activation context for use when loading extensions. diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props --- a/PCbuild/pyproject.props +++ b/PCbuild/pyproject.props @@ -87,7 +87,7 @@ diff --git a/PCbuild/python.props b/PCbuild/python.props --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -1,7 +1,7 @@ - + - Win32 + Win32 Release .cp$(MajorVersionNumber)$(MinorVersionNumber)-win32 .cp$(MajorVersionNumber)$(MinorVersionNumber)-win_amd64 + + + $(MajorVersionNumber).$(MinorVersionNumber) + $(SysWinVer)-32 diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -67,7 +67,7 @@ /Zm200 %(AdditionalOptions) $(PySourcePath)Python;$(PySourcePath)Modules\zlib;%(AdditionalIncludeDirectories) - _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;%(PreprocessorDefinitions) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;MS_DLL_ID="$(SysWinVer)";%(PreprocessorDefinitions) ws2_32.lib;%(AdditionalDependencies) @@ -335,7 +335,6 @@ - @@ -387,13 +386,16 @@ + + + - + -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 18:12:45 2015 From: python-checkins at python.org (steve.dower) Date: Thu, 15 Jan 2015 17:12:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_23018=3A_Add_version?= =?utf-8?q?_info_to_python=5Bw=5D=2Eexe?= Message-ID: <20150115171218.72555.4931@psf.io> https://hg.python.org/cpython/rev/3f7e483cebef changeset: 94180:3f7e483cebef user: Steve Dower date: Thu Jan 15 09:10:43 2015 -0800 summary: Issue 23018: Add version info to python[w].exe files: PC/python.manifest | 12 ++++++++ PC/python_exe.rc | 50 +++++++++++++++++++++++++++++++++- PC/python_nt.rc | 42 ++++++--------------------- PC/python_ver_rc.h | 35 +++++++++++++++++++++++ 4 files changed, 106 insertions(+), 33 deletions(-) diff --git a/PC/python.manifest b/PC/python.manifest new file mode 100644 --- /dev/null +++ b/PC/python.manifest @@ -0,0 +1,12 @@ + + + + + + + + + + + + \ No newline at end of file diff --git a/PC/python_exe.rc b/PC/python_exe.rc --- a/PC/python_exe.rc +++ b/PC/python_exe.rc @@ -1,1 +1,49 @@ -1 ICON DISCARDABLE "pycon.ico" +// Resource script for Python console EXEs. + +#include "python_ver_rc.h" + +// Include the manifest file that indicates we support all +// current versions of Windows. +#include +1 RT_MANIFEST "python.manifest" + +1 ICON DISCARDABLE "pycon.ico" + + +///////////////////////////////////////////////////////////////////////////// +// +// Version +// + +VS_VERSION_INFO VERSIONINFO + FILEVERSION PYVERSION64 + PRODUCTVERSION PYVERSION64 + FILEFLAGSMASK 0x3fL +#ifdef _DEBUG + FILEFLAGS VS_FF_DEBUG +#else + FILEFLAGS 0x0L +#endif + FILEOS VOS__WINDOWS32 + FILETYPE VFT_APP + FILESUBTYPE 0x0L +BEGIN + BLOCK "StringFileInfo" + BEGIN + BLOCK "000004b0" + BEGIN + VALUE "CompanyName", PYTHON_COMPANY "\0" + VALUE "FileDescription", "Python\0" + VALUE "FileVersion", PYTHON_VERSION + VALUE "InternalName", "Python Console\0" + VALUE "LegalCopyright", PYTHON_COPYRIGHT "\0" + VALUE "OriginalFilename", "python" PYTHON_DEBUG_EXT ".exe\0" + VALUE "ProductName", "Python\0" + VALUE "ProductVersion", PYTHON_VERSION + END + END + BLOCK "VarFileInfo" + BEGIN + VALUE "Translation", 0x0, 1200 + END +END diff --git a/PC/python_nt.rc b/PC/python_nt.rc --- a/PC/python_nt.rc +++ b/PC/python_nt.rc @@ -1,33 +1,11 @@ // Resource script for Python core DLL. -// Currently only holds version information. -// -#include "winver.h" -#define MS_WINDOWS -#include "modsupport.h" -#include "patchlevel.h" -#ifdef _DEBUG -# include "pythonnt_rc_d.h" -#else -# include "pythonnt_rc.h" -#endif +#include "python_ver_rc.h" -/* e.g., 3.3.0a1 - * PY_VERSION comes from patchlevel.h - */ -#define PYTHON_VERSION PY_VERSION "\0" - -/* 64-bit version number as comma-separated list of 4 16-bit ints */ -#if PY_MICRO_VERSION > 64 -# error "PY_MICRO_VERSION > 64" -#endif -#if PY_RELEASE_LEVEL > 99 -# error "PY_RELEASE_LEVEL > 99" -#endif -#if PY_RELEASE_SERIAL > 9 -# error "PY_RELEASE_SERIAL > 9" -#endif -#define PYVERSION64 PY_MAJOR_VERSION, PY_MINOR_VERSION, FIELD3, PYTHON_API_VERSION +// Include the manifest file that indicates we support all +// current versions of Windows. +#include +2 RT_MANIFEST "python.manifest" // String Tables STRINGTABLE DISCARDABLE @@ -45,23 +23,23 @@ PRODUCTVERSION PYVERSION64 FILEFLAGSMASK 0x3fL #ifdef _DEBUG - FILEFLAGS 0x1L + FILEFLAGS VS_FF_DEBUG #else FILEFLAGS 0x0L #endif - FILEOS 0x40004L - FILETYPE 0x1L + FILEOS VOS__WINDOWS32 + FILETYPE VFT_DLL FILESUBTYPE 0x0L BEGIN BLOCK "StringFileInfo" BEGIN BLOCK "000004b0" BEGIN - VALUE "CompanyName", "Python Software Foundation\0" + VALUE "CompanyName", PYTHON_COMPANY "\0" VALUE "FileDescription", "Python Core\0" VALUE "FileVersion", PYTHON_VERSION VALUE "InternalName", "Python DLL\0" - VALUE "LegalCopyright", "Copyright ? 2001-2015 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC.\0" + VALUE "LegalCopyright", PYTHON_COPYRIGHT "\0" VALUE "OriginalFilename", PYTHON_DLL_NAME "\0" VALUE "ProductName", "Python\0" VALUE "ProductVersion", PYTHON_VERSION diff --git a/PC/python_ver_rc.h b/PC/python_ver_rc.h new file mode 100644 --- /dev/null +++ b/PC/python_ver_rc.h @@ -0,0 +1,35 @@ +// Resource script for Python core DLL. +// Currently only holds version information. +// +#include "winver.h" + +#define PYTHON_COMPANY "Python Software Foundation" +#define PYTHON_COPYRIGHT "Copyright ? 2001-2014 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC." + +#define MS_WINDOWS +#include "modsupport.h" +#include "patchlevel.h" +#ifdef _DEBUG +# include "pythonnt_rc_d.h" +# define PYTHON_DEBUG_EXT "_d" +#else +# include "pythonnt_rc.h" +# define PYTHON_DEBUG_EXT +#endif + +/* e.g., 3.3.0a1 + * PY_VERSION comes from patchlevel.h + */ +#define PYTHON_VERSION PY_VERSION "\0" + +/* 64-bit version number as comma-separated list of 4 16-bit ints */ +#if PY_MICRO_VERSION > 64 +# error "PY_MICRO_VERSION > 64" +#endif +#if PY_RELEASE_LEVEL > 99 +# error "PY_RELEASE_LEVEL > 99" +#endif +#if PY_RELEASE_SERIAL > 9 +# error "PY_RELEASE_SERIAL > 9" +#endif +#define PYVERSION64 PY_MAJOR_VERSION, PY_MINOR_VERSION, FIELD3, PYTHON_API_VERSION -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 18:18:33 2015 From: python-checkins at python.org (steve.dower) Date: Thu, 15 Jan 2015 17:18:33 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMyMzE2?= =?utf-8?q?0=3A_Respect_the_environment_variable_SVNROOT_in_external-commo?= =?utf-8?q?n=2Ebat?= Message-ID: <20150115171819.11571.5114@psf.io> https://hg.python.org/cpython/rev/294501835890 changeset: 94181:294501835890 branch: 2.7 parent: 94157:25a1ce2a6f9b user: Steve Dower date: Thu Jan 15 09:15:45 2015 -0800 summary: Closes #23160: Respect the environment variable SVNROOT in external-common.bat (patch by anselm.kruis) files: Tools/buildbot/external-common.bat | 18 ++++++++++-------- 1 files changed, 10 insertions(+), 8 deletions(-) diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -1,6 +1,8 @@ @rem Common file shared between external.bat and external-amd64.bat. Responsible for @rem fetching external components into the root\externals directory. +if "%SVNROOT%"=="" set SVNROOT=http://svn.python.org/projects/external/ + if not exist externals mkdir externals cd externals @rem XXX: If you need to force the buildbots to start from a fresh environment, uncomment @@ -31,31 +33,31 @@ @rem bzip if not exist bzip2-1.0.6 ( rd /s/q bzip2-1.0.5 - svn export http://svn.python.org/projects/external/bzip2-1.0.6 + svn export %SVNROOT%bzip2-1.0.6 ) @rem Berkeley DB if exist db-4.4.20 rd /s/q db-4.4.20 -if not exist db-4.7.25.0 svn export http://svn.python.org/projects/external/db-4.7.25.0 +if not exist db-4.7.25.0 svn export %SVNROOT%db-4.7.25.0 @rem NASM, for OpenSSL build @rem if exist nasm-2.11.06 rd /s/q nasm-2.11.06 -if not exist nasm-2.11.06 svn export http://svn.python.org/projects/external/nasm-2.11.06 +if not exist nasm-2.11.06 svn export %SVNROOT%nasm-2.11.06 @rem OpenSSL if exist openssl-1.0.1i rd /s/q openssl-1.0.1i -if not exist openssl-1.0.1j svn export http://svn.python.org/projects/external/openssl-1.0.1j +if not exist openssl-1.0.1j svn export %SVNROOT%openssl-1.0.1j @rem tcl/tk if not exist tcl-8.5.15.0 ( rd /s/q tcltk tcltk64 tcl-8.5.2.1 tk-8.5.2.0 - svn export http://svn.python.org/projects/external/tcl-8.5.15.0 + svn export %SVNROOT%tcl-8.5.15.0 ) -if not exist tk-8.5.15.0 svn export http://svn.python.org/projects/external/tk-8.5.15.0 -if not exist tix-8.4.3.5 svn export http://svn.python.org/projects/external/tix-8.4.3.5 +if not exist tk-8.5.15.0 svn export %SVNROOT%tk-8.5.15.0 +if not exist tix-8.4.3.5 svn export %SVNROOT%tix-8.4.3.5 @rem sqlite3 if not exist sqlite-3.6.21 ( rd /s/q sqlite-source-3.3.4 - svn export http://svn.python.org/projects/external/sqlite-3.6.21 + svn export %SVNROOT%sqlite-3.6.21 ) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 18:18:45 2015 From: python-checkins at python.org (steve.dower) Date: Thu, 15 Jan 2015 17:18:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_from_3=2E4?= Message-ID: <20150115171837.22425.5221@psf.io> https://hg.python.org/cpython/rev/c52f76c84eec changeset: 94183:c52f76c84eec parent: 94180:3f7e483cebef parent: 94182:45e2c95bb802 user: Steve Dower date: Thu Jan 15 09:17:18 2015 -0800 summary: Null merge from 3.4 files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 18:18:45 2015 From: python-checkins at python.org (steve.dower) Date: Thu, 15 Jan 2015 17:18:45 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMzE2?= =?utf-8?q?0=3A_Respect_the_environment_variable_SVNROOT_in_external-commo?= =?utf-8?q?n=2Ebat?= Message-ID: <20150115171837.11561.7624@psf.io> https://hg.python.org/cpython/rev/45e2c95bb802 changeset: 94182:45e2c95bb802 branch: 3.4 parent: 94177:8adf1896712d user: Steve Dower date: Thu Jan 15 09:16:38 2015 -0800 summary: Closes #23160: Respect the environment variable SVNROOT in external-common.bat (patch by anselm.kruis) files: Tools/buildbot/external-common.bat | 18 ++++++++++-------- 1 files changed, 10 insertions(+), 8 deletions(-) diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -1,6 +1,8 @@ @rem Common file shared between external.bat and external-amd64.bat. Responsible for @rem fetching external components into the root\.. buildbot directories. +if "%SVNROOT%"=="" set SVNROOT=http://svn.python.org/projects/external/ + if not exist externals mkdir externals cd externals @rem XXX: If you need to force the buildbots to start from a fresh environment, uncomment @@ -18,35 +20,35 @@ @rem bzip if not exist bzip2-1.0.6 ( rd /s/q bzip2-1.0.5 - svn export http://svn.python.org/projects/external/bzip2-1.0.6 + svn export %SVNROOT%bzip2-1.0.6 ) @rem NASM, for OpenSSL build @rem if exist nasm-2.11.06 rd /s/q nasm-2.11.06 -if not exist nasm-2.11.06 svn export http://svn.python.org/projects/external/nasm-2.11.06 +if not exist nasm-2.11.06 svn export %SVNROOT%nasm-2.11.06 @rem OpenSSL if not exist openssl-1.0.1j ( rd /s/q openssl-1.0.1i - svn export http://svn.python.org/projects/external/openssl-1.0.1j + svn export %SVNROOT%openssl-1.0.1j ) @rem tcl/tk if not exist tcl-8.6.1.0 ( rd /s/q tcltk tcltk64 tcl-8.5.11.0 tk-8.5.11.0 - svn export http://svn.python.org/projects/external/tcl-8.6.1.0 + svn export %SVNROOT%tcl-8.6.1.0 ) -if not exist tk-8.6.1.0 svn export http://svn.python.org/projects/external/tk-8.6.1.0 -if not exist tix-8.4.3.4 svn export http://svn.python.org/projects/external/tix-8.4.3.4 +if not exist tk-8.6.1.0 svn export %SVNROOT%tk-8.6.1.0 +if not exist tix-8.4.3.4 svn export %SVNROOT%tix-8.4.3.4 @rem sqlite3 if not exist sqlite-3.8.3.1 ( rd /s/q sqlite-source-3.8.1 - svn export http://svn.python.org/projects/external/sqlite-3.8.3.1 + svn export %SVNROOT%sqlite-3.8.3.1 ) @rem lzma if not exist xz-5.0.5 ( rd /s/q xz-5.0.3 - svn export http://svn.python.org/projects/external/xz-5.0.5 + svn export %SVNROOT%xz-5.0.5 ) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 21:52:24 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 20:52:24 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Backout_change?= =?utf-8?q?set_6ab2575bc12b?= Message-ID: <20150115205216.104134.94410@psf.io> https://hg.python.org/cpython/rev/ecbde0b31f6f changeset: 94184:ecbde0b31f6f branch: 3.4 parent: 94182:45e2c95bb802 user: Victor Stinner date: Thu Jan 15 21:50:19 2015 +0100 summary: Backout changeset 6ab2575bc12b StreamWriter: close() now clears the reference to the transport StreamWriter now raises an exception if it is closed: write(), writelines(), write_eof(), can_write_eof(), get_extra_info(), drain(). files: Lib/asyncio/streams.py | 25 ++++--------------------- 1 files changed, 4 insertions(+), 21 deletions(-) diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -258,22 +258,8 @@ self._reader = reader self._loop = loop - def close(self): - if self._transport is None: - return - self._transport.close() - self._transport = None - - def _check_closed(self): - if self._transport is None: - raise RuntimeError('StreamWriter is closed') - def __repr__(self): - info = [self.__class__.__name__] - if self._transport is not None: - info.append('transport=%r' % self._transport) - else: - info.append('closed') + info = [self.__class__.__name__, 'transport=%r' % self._transport] if self._reader is not None: info.append('reader=%r' % self._reader) return '<%s>' % ' '.join(info) @@ -283,23 +269,21 @@ return self._transport def write(self, data): - self._check_closed() self._transport.write(data) def writelines(self, data): - self._check_closed() self._transport.writelines(data) def write_eof(self): - self._check_closed() return self._transport.write_eof() def can_write_eof(self): - self._check_closed() return self._transport.can_write_eof() + def close(self): + return self._transport.close() + def get_extra_info(self, name, default=None): - self._check_closed() return self._transport.get_extra_info(name, default) @coroutine @@ -311,7 +295,6 @@ w.write(data) yield from w.drain() """ - self._check_closed() if self._reader is not None: exc = self._reader.exception() if exc is not None: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 21:52:24 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 20:52:24 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150115205216.93201.649@psf.io> https://hg.python.org/cpython/rev/973173ad0fe7 changeset: 94185:973173ad0fe7 parent: 94183:c52f76c84eec parent: 94184:ecbde0b31f6f user: Victor Stinner date: Thu Jan 15 21:50:48 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/streams.py | 25 ++++--------------------- 1 files changed, 4 insertions(+), 21 deletions(-) diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -258,22 +258,8 @@ self._reader = reader self._loop = loop - def close(self): - if self._transport is None: - return - self._transport.close() - self._transport = None - - def _check_closed(self): - if self._transport is None: - raise RuntimeError('StreamWriter is closed') - def __repr__(self): - info = [self.__class__.__name__] - if self._transport is not None: - info.append('transport=%r' % self._transport) - else: - info.append('closed') + info = [self.__class__.__name__, 'transport=%r' % self._transport] if self._reader is not None: info.append('reader=%r' % self._reader) return '<%s>' % ' '.join(info) @@ -283,23 +269,21 @@ return self._transport def write(self, data): - self._check_closed() self._transport.write(data) def writelines(self, data): - self._check_closed() self._transport.writelines(data) def write_eof(self): - self._check_closed() return self._transport.write_eof() def can_write_eof(self): - self._check_closed() return self._transport.can_write_eof() + def close(self): + return self._transport.close() + def get_extra_info(self, name, default=None): - self._check_closed() return self._transport.get_extra_info(name, default) @coroutine @@ -311,7 +295,6 @@ w.write(data) yield from w.drain() """ - self._check_closed() if self._reader is not None: exc = self._reader.exception() if exc is not None: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 23:01:28 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 22:01:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150115215845.93203.91300@psf.io> https://hg.python.org/cpython/rev/031fc0231f3d changeset: 94187:031fc0231f3d parent: 94185:973173ad0fe7 parent: 94186:992ce0dcfb29 user: Victor Stinner date: Thu Jan 15 22:53:21 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/test/test_asyncio/test_subprocess.py | 14 ++++++++++- 1 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -179,6 +179,18 @@ 'sys.stdout.write("x" * %s)' % size, 'sys.stdout.flush()', )) + + connect_read_pipe = self.loop.connect_read_pipe + + @asyncio.coroutine + def connect_read_pipe_mock(*args, **kw): + transport, protocol = yield from connect_read_pipe(*args, **kw) + transport.pause_reading = mock.Mock() + transport.resume_reading = mock.Mock() + return (transport, protocol) + + self.loop.connect_read_pipe = connect_read_pipe_mock + proc = yield from asyncio.create_subprocess_exec( sys.executable, '-c', code, stdin=asyncio.subprocess.PIPE, @@ -186,8 +198,6 @@ limit=limit, loop=self.loop) stdout_transport = proc._transport.get_pipe_transport(1) - stdout_transport.pause_reading = mock.Mock() - stdout_transport.resume_reading = mock.Mock() stdout, stderr = yield from proc.communicate() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 15 23:01:28 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 15 Jan 2015 22:01:28 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNjg1?= =?utf-8?q?=3A_Fix_test=5Fpause=5Freading=28=29_of_asyncio/test=5Fsubproce?= =?utf-8?q?ss?= Message-ID: <20150115215845.103313.99238@psf.io> https://hg.python.org/cpython/rev/992ce0dcfb29 changeset: 94186:992ce0dcfb29 branch: 3.4 parent: 94184:ecbde0b31f6f user: Victor Stinner date: Thu Jan 15 22:52:59 2015 +0100 summary: Issue #22685: Fix test_pause_reading() of asyncio/test_subprocess Override the connect_read_pipe() method of the loop to mock immediatly pause_reading() and resume_reading() methods. The test failed randomly on FreeBSD 9 buildbot and on Windows using trollius. files: Lib/test/test_asyncio/test_subprocess.py | 14 ++++++++++- 1 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -179,6 +179,18 @@ 'sys.stdout.write("x" * %s)' % size, 'sys.stdout.flush()', )) + + connect_read_pipe = self.loop.connect_read_pipe + + @asyncio.coroutine + def connect_read_pipe_mock(*args, **kw): + transport, protocol = yield from connect_read_pipe(*args, **kw) + transport.pause_reading = mock.Mock() + transport.resume_reading = mock.Mock() + return (transport, protocol) + + self.loop.connect_read_pipe = connect_read_pipe_mock + proc = yield from asyncio.create_subprocess_exec( sys.executable, '-c', code, stdin=asyncio.subprocess.PIPE, @@ -186,8 +198,6 @@ limit=limit, loop=self.loop) stdout_transport = proc._transport.get_pipe_transport(1) - stdout_transport.pause_reading = mock.Mock() - stdout_transport.resume_reading = mock.Mock() stdout, stderr = yield from proc.communicate() -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Fri Jan 16 09:05:59 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 16 Jan 2015 09:05:59 +0100 Subject: [Python-checkins] Daily reference leaks (031fc0231f3d): sum=6 Message-ID: results for 031fc0231f3d on branch "default" -------------------------------------------- test_asyncio leaked [3, 0, 0] memory blocks, sum=3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogeikkXn', '-x'] From python-checkins at python.org Fri Jan 16 18:05:23 2015 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 16 Jan 2015 17:05:23 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_First_public_draft_of_PEP_484?= Message-ID: <20150116170520.93203.12248@psf.io> https://hg.python.org/peps/rev/0dd72b08606a changeset: 5673:0dd72b08606a user: Guido van Rossum date: Fri Jan 16 09:05:19 2015 -0800 summary: First public draft of PEP 484 files: pep-0484.txt | 484 ++++++++++++++++++++++++++++++++++++++- 1 files changed, 479 insertions(+), 5 deletions(-) diff --git a/pep-0484.txt b/pep-0484.txt --- a/pep-0484.txt +++ b/pep-0484.txt @@ -3,19 +3,493 @@ Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum , Jukka Lehtosalo , ?ukasz Langa -Discussions-To: Python-Ideas +Discussions-To: Python-Dev Status: Draft Type: Standards Track Content-Type: text/x-rst -Created: 08-Jan-2015 -Post-History: +Created: 29-Sep-2014 +Post-History: 16-Jan-2015 Resolution: + Abstract ======== -This PEP is currently a stub. The content should be copied from -https://github.com/ambv/typehinting (but omitting the literature overview, which is PEP 482) and reformatted. +This PEP introduces a standard syntax for type hints using annotations +on function definitions. + +The proposal is strongly inspired by mypy [mypy]_. + + +Rationale and Goals +=================== + +PEP 3107 added support for arbitrary annotations on parts of a function +definition. Although no meaning was assigned to annotations then, there +has always been an implicit goal to use them for type hinting, which is +listed as the first possible use case in said PEP. + +This PEP aims to provide a standard syntax for type annotations, opening +up Python code to easier static analysis and refactoring, potential +runtime type checking, and performance optimizations utilizing type +information. + + +Type Definition Syntax +====================== + +The syntax leverages PEP 3107-style annotations with a number of +extensions described in sections below. In its basic form, type hinting +is used by filling function annotations with classes:: + + def greeting(name: str) -> str: + return 'Hello ' + name + +This denotes that the expected type of the ``name`` argument is ``str``. +Analogically, the expected return type is ``str``. Subclasses of +a specified argument type are also accepted as valid types for that +argument. + +Abstract base classes, types available in the ``types`` module, and +user-defined classes may be used as type hints as well. Annotations +must be valid expressions that evaluate without raising exceptions at +the time the function is defined. In addition, the needs of static +analysis require that annotations must be simple enough to be +interpreted by static analysis tools. (This is an intentionally +somewhat vague requirement.) + +.. FIXME: Define rigorously what is/isn't supported. + +When used as an annotation, the expression ``None`` is considered +equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting +purposes. + +Type aliases are also valid type hints:: + + integer = int + + def retry(url: str, retry_count: integer): ... + +New names that are added to support features described in following +sections are available in the ``typing`` package. + + +Callbacks +--------- + +Frameworks expecting callback functions of specific signatures might be +type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. +Examples:: + + from typing import Any, AnyArgs, Callable + + def feeder(get_next_item: Callable[[], Item]): ... + + def async_query(on_success: Callable[[int], None], on_error: Callable[[int, Exception], None]): ... + + def partial(func: Callable[AnyArgs, Any], *args): ... + +Since using callbacks with keyword arguments is not perceived as +a common use case, there is currently no support for specifying keyword +arguments with ``Callable``. + + +Generics +-------- + +Since type information about objects kept in containers cannot be +statically inferred in a generic way, abstract base classes have been +extended to support subscription to denote expected types for container +elements. Example:: + + from typing import Mapping, Set + + def notify_by_email(employees: Set[Employee], overrides: Mapping[str, str]): ... + +Generics can be parametrized by using a new factory available in +``typing`` called ``TypeVar``. Example:: + + from typing import Sequence, TypeVar + + T = TypeVar('T') # Declare type variable + + def first(l: Sequence[T]) -> T: # Generic function + return l[0] + +In this case the contract is that the returning value is consistent with +the elements held by the collection. + +``TypeVar`` supports constraining parametric types to classes with any of +the specified bases. Example:: + + from typing import Iterable + + X = TypeVar('X') + Y = TypeVar('Y', Iterable[X]) + + def filter(rule: Callable[[X], bool], input: Y) -> Y: + ... + +.. FIXME: Add an example with multiple bases defined. + +In the example above we specify that ``Y`` can be any subclass of +Iterable with elements of type ``X``, as long as the return type of +``filter()`` will be the same as the type of the ``input`` +argument. + +.. FIXME: Explain more about how this works. + + +Forward references +------------------ + +When a type hint contains names that have not been defined yet, that +definition may be expressed as a string, to be resolved later. For +example, instead of writing:: + + def notify_by_email(employees: Set[Employee]): ... + +one might write:: + + def notify_by_email(employees: 'Set[Employee]'): ... + +.. FIXME: Rigorously define this. Defend it, or find an alternative. + + +Union types +----------- + +Since accepting a small, limited set of expected types for a single +argument is common, there is a new special factory called ``Union``. +Example:: + + from typing import Union + + def handle_employees(e: Union[Employee, Sequence[Employee]]): + if isinstance(e, Employee): + e = [e] + ... + +A type factored by ``Union[T1, T2, ...]`` responds ``True`` to +``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and +any of its subclasses, and so on. + +One common case of union types are *optional* types. By default, +``None`` is an invalid value for any type, unless a default value of +``None`` has been provided in the function definition. Examples:: + + def handle_employee(e: Union[Employee, None]): ... + +As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; +for example, the above is equivalent to:: + + from typing import Optional + + def handle_employee(e: Optional[Employee]): ... + +An optional type is also automatically assumed when the default value is +``None``, for example:: + + def handle_employee(e: Employee = None): ... + +This is equivalent to:: + + def handle_employee(e: Optional[Employee] = None): ... + +.. FIXME: Is this really a good idea? + +A special kind of union type is ``Any``, a class that responds +``True`` to ``issubclass`` of any class. This lets the user +explicitly state that there are no constraints on the type of a +specific argument or return value. + + +Platform-specific type checking +------------------------------- + +In some cases the typing information will depend on the platform that +the program is being executed on. To enable specifying those +differences, simple conditionals can be used:: + + from typing import PY2, WINDOWS + + if PY2: + text = unicode + else: + text = str + + def f() -> text: ... + + if WINDOWS: + loop = ProactorEventLoop + else: + loop = UnixSelectorEventLoop + +Arbitrary literals defined in the form of ``NAME = True`` will also be +accepted by the type checker to differentiate type resolution:: + + DEBUG = False + ... + if DEBUG: + class Tracer: + + else: + class Tracer: + + +For the purposes of type hinting, the type checker assumes ``__debug__`` +is set to ``True``, in other words the ``-O`` command-line option is not +used while type checking. + + +Compatibility with other uses of function annotations +----------------------------------------------------- + +A number of existing or potential use cases for function annotations +exist, which are incompatible with type hinting. These may confuse a +static type checker. However, since type hinting annotations have no +run time behavior (other than evaluation of the annotation expression +and storing annotations in the ``__annotations__`` attribute of the +function object), this does not make the program incorrect -- it just +makes it issue warnings when a static analyzer is used. + +To mark portions of the program that should not be covered by type +hinting, use the following: + +* a ``@no_type_checks`` decorator on classes and functions + +* a ``# type: ignore`` comment on arbitrary lines + +.. FIXME: should we have a module-wide comment as well? + + +Type Hints on Local and Global Variables +======================================== + +No first-class syntax support for explicitly marking variables as being +of a specific type is added by this PEP. To help with type inference in +complex cases, a comment of the following format may be used:: + + x = [] # type: List[Employee] + +In the case where type information for a local variable is needed before +if was declared, an ``Undefined`` placeholder might be used:: + + from typing import Undefined + + x = Undefined # type: List[Employee] + y = Undefined(int) + +If type hinting proves useful in general, a syntax for typing variables +may be provided in a future Python version. + + +Explicit raised exceptions +========================== + +No support for listing explicitly raised exceptions is being defined by +this PEP. Currently the only known use case for this feature is +documentational, in which case the recommendation is to put this +information in a docstring. + + +The ``typing`` package +====================== + +To open the usage of static type checking to Python 3.5 as well as older +versions, a uniform namespace is required. For this purpose, a new +package in the standard library is introduced called ``typing``. It +holds a set of classes representing builtin types with generics, namely: + +* Dict, used as ``Dict[key_type, value_type]`` + +* List, used as ``List[element_type]`` + +* Set, used as ``Set[element_type]``. See remark for ``AbstractSet`` + below. + +* FrozenSet, used as ``FrozenSet[element_type]`` + +* Tuple, used as ``Tuple[index0_type, index1_type, ...]``. + Arbitrary-length tuples might be expressed using ellipsis, in which + case the following arguments are considered the same type as the last + defined type on the tuple. + +It also introduces factories and helper members needed to express +generics and union types: + +* Any, used as ``def get(key: str) -> Any: ...`` + +* Union, used as ``Union[Type1, Type2, Type3]`` + +* TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply + ``Y = TypeVar('Y')`` + +* Undefined, used as ``local_variable = Undefined # type: List[int]`` or + ``local_variable = Undefined(List[int])`` (the latter being slower + during runtime) + +* Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]`` + +* AnyArgs, used as ``Callable[AnyArgs, ReturnType]`` + +* AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)`` + +All abstract base classes available in ``collections.abc`` are +importable from the ``typing`` package, with added generics support: + +* ByteString + +* Callable + +* Container + +* Hashable + +* ItemsView + +* Iterable + +* Iterator + +* KeysView + +* Mapping + +* MappingView + +* MutableMapping + +* MutableSequence + +* MutableSet + +* Sequence + +* Set as ``AbstractSet``. This name change was required because ``Set`` + in the ``typing`` module means ``set()`` with generics. + +* Sized + +* ValuesView + +* Mapping + +The library includes literals for platform-specific type hinting: + +* PY2 + +* PY3, equivalent to ``not PY2`` + +* WINDOWS + +* UNIXOID, equivalent to ``not WINDOWS`` + +The following types are available in the ``typing.io`` module: + +* IO + +* BinaryIO + +* TextIO + +The following types are provided by the ``typing.re`` module: + +* Match and Pattern, types of ``re.match()`` and ``re.compile()`` + results + +As a convenience measure, types from ``typing.io`` and ``typing.re`` are +also available in ``typing`` (quoting Guido, "There's a reason those +modules have two-letter names."). + + +The place of the ``typing`` module in the standard library +---------------------------------------------------------- + +.. FIXME: complete this section + + +Usage Patterns +============== + +The main use case of type hinting is static analysis using an external +tool without executing the analyzed program. Existing tools used for +that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_ +might be extended to support type checking. New tools, like mypy's +``mypy -S`` mode, can be adopted specifically for this purpose. + +Type checking based on type hints is understood as a best-effort +mechanism. In other words, whenever types are not annotated and cannot +be inferred, the type checker considers such code valid. Type errors +are only reported in case of explicit or inferred conflict. Moreover, +as a mechanism that is not tied to execution of the code, it does not +affect runtime behaviour. In other words, even in the case of a typing +error, the program will continue running. + +The implementation of a type checker, whether linting source files or +enforcing type information during runtime, is out of scope for this PEP. + +.. FIXME: Describe stub modules. + +.. FIXME: Describe run-time behavior of generic types. + + +Existing Approaches +=================== + +PEP 482 lists existing approaches in Python and other languages. + + +Is type hinting Pythonic? +========================= + +Type annotations provide important documentation for how a unit of code +should be used. Programmers should therefore provide type hints on +public APIs, namely argument and return types on functions and methods +considered public. However, because types of local and global variables +can be often inferred, they are rarely necessary. + +The kind of information that type hints hold has always been possible to +achieve by means of docstrings. In fact, a number of formalized +mini-languages for describing accepted arguments have evolved. Moving +this information to the function declaration makes it more visible and +easier to access both at runtime and by static analysis. Adding to that +the notion that ?explicit is better than implicit?, type hints are +indeed *Pythonic*. + + +Acknowledgements +================ + +This document could not be completed without valuable input, +encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson +Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski. + +Influences include existing languages, libraries and frameworks +mentioned in PEP 482. Many thanks to their creators, in alphabetical +order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings, +Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer, +Raoul-Gabriel Urma, and Julien Verlaguet. + + +References +========== + +.. [mypy] + http://mypy-lang.org + +.. [pyflakes] + https://github.com/pyflakes/pyflakes/ + +.. [pylint] + http://www.pylint.org + + +Copyright +========= + +This document has been placed in the public domain. + .. -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 16 18:11:30 2015 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 16 Jan 2015 17:11:30 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_48=7B2=2C3=7D_title_chang?= =?utf-8?q?es_=28typing_-=3E_types=29_and_add_some_cross-links=2E?= Message-ID: <20150116171108.93189.30051@psf.io> https://hg.python.org/peps/rev/3e552a15fb9f changeset: 5674:3e552a15fb9f user: Guido van Rossum date: Fri Jan 16 09:11:05 2015 -0800 summary: PEP 48{2,3} title changes (typing -> types) and add some cross-links. files: pep-0482.txt | 4 ++-- pep-0483.txt | 4 ++-- pep-0484.txt | 2 ++ 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/pep-0482.txt b/pep-0482.txt --- a/pep-0482.txt +++ b/pep-0482.txt @@ -1,5 +1,5 @@ PEP: 482 -Title: Literature Overview for Type Hinting +Title: Literature Overview for Type Hints Version: $Revision$ Last-Modified: $Date$ Author: ?ukasz Langa @@ -15,7 +15,7 @@ ======== This PEP is one of three related to type hinting. This PEP gives a -literature overview of related work. +literature overview of related work. The main spec is PEP 484. Existing Approaches for Python diff --git a/pep-0483.txt b/pep-0483.txt --- a/pep-0483.txt +++ b/pep-0483.txt @@ -1,5 +1,5 @@ PEP: 483 -Title: The Theory of Type Hinting +Title: The Theory of Type Hints Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum @@ -14,7 +14,7 @@ Abstract ======== -This PEP lays out the theory to be referenced by PEP 484. +This PEP lays out the theory referenced by PEP 484. Introduction diff --git a/pep-0484.txt b/pep-0484.txt --- a/pep-0484.txt +++ b/pep-0484.txt @@ -20,6 +20,8 @@ The proposal is strongly inspired by mypy [mypy]_. +The theory behind type hints and gradual typing is explained in PEP 483. + Rationale and Goals =================== -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 16 18:59:40 2015 From: python-checkins at python.org (brett.cannon) Date: Fri, 16 Jan 2015 17:59:40 +0000 Subject: [Python-checkins] =?utf-8?q?devguide=3A_Issue_=2322992=3A_A_git_d?= =?utf-8?q?eveloper=27s_guide_to_hg=2E?= Message-ID: <20150116175929.93179.52186@psf.io> https://hg.python.org/devguide/rev/51c89a0c30b4 changeset: 726:51c89a0c30b4 user: Brett Cannon date: Fri Jan 16 12:59:22 2015 -0500 summary: Issue #22992: A git developer's guide to hg. Thanks to Demian Brecht for the guide! files: gitdevs.rst | 509 ++++++++++++++++++++++++++++++++++++++++ index.rst | 3 + 2 files changed, 512 insertions(+), 0 deletions(-) diff --git a/gitdevs.rst b/gitdevs.rst new file mode 100644 --- /dev/null +++ b/gitdevs.rst @@ -0,0 +1,509 @@ +:tocdepth: 4 + +.. highlight:: bash +.. _gitdevs: + +Mercurial for git developers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. contents:: + :local: + + +Overview +======== + +This guide is geared towards prospective contributors who are accustomed to +git-based development. The intention of this guide is to lower the initial +barrier of entry for prospective contributors in learning a new source control +tool. While there are projects that focus on automating this to allow seamless +bridges between the two systems, this guide focusses solely on native Mercurial +functionality. As with most tools, this is not the only method of achieving +git-like workflows with Mercurial. It is intended solely to provide one +potential path. + +Note that this geared towards prospective contributors and the patch workflow. +Core developers will have a similar workflow, but it will differ in the later +stages (merge vs patch generation). + +The workflow that will be used as the case study for this guide is as follows: + +* Clone CPython +* Find/Create issue in the issue tracker (issueA) + + - Work on issue + - Submit patch for review + +* Work on another issue (issueB) + + - Work on issue + - Commit work in progress + +* Address review comments (issueA) + + - Commit work + - Submit patch for review + +* Continue feature work (issueB) +* *Rinse and repeat* + + +Git workflow +============ + +With the above workflow in mind, simplified steps using git may look like +this:: + + git clone git at github.com:python/cpython.git + + # work on issueA + git checkout -b issueA + git commit -a + + # start working on issueB + git checkout master + git checkout -b issueB + git commit -a + + # address review comments + git checkout issueA + git commit -a + + # optionally rebase work + git pull --rebase + + # generate patch for submission + git diff master..issueA > issueA.patch + + # continue working on issueB + git checkout issueB + + +This flow will be used as the base for comparison against Mercurial usage that +aims to achieve a similar, if not identical, workflow. + + +Main differences between git and hg +=================================== + +This section aims to list major differences that may be unexpected by a git +developer. While there are many differences between git and Mercurial, only +those that are immediately relevant to new users coming from git are examined. + + +Mercurial branches are global and permanent +------------------------------------------- + +This means that if you create a named branch, it is intended to be long-lived: + + Mercurial branch names may be used for whatever reasons users want. + However, a good rule of thumb is to use branch names sparingly and for + rather longer lived concepts like "release branches" (rel-1, rel-2, etc) + and rather not for short lived work of single developers. + + -- http://mercurial.selenic.com/wiki/Branch#Named_branches + + +If you routinely create short-lived branches for development work and then +delete them after they've been merged to master, this is something to be aware +of. You *can* still only push select branches to remotes, so it's not a +functional issue as far as the public repo goes. + + +Workflow options using Mercurial +-------------------------------- + +There are a number of paths that could be taken to achieve a sane development +workflow. Some of these are: + + +Named branches +'''''''''''''' + + Mercurial supports giving names to branches, by using the branch name + property of the changeset (see NamedBranches_). If no branch name was set, + Mercurial assigns the branch name "default". So the name of the default + branch in a repository is "default" (which, for example, is not displayed + when doing a hg log). + + Unfortunately, as Mercurial named (and default) branches are global and + permanent, they don't lend themselves well to local development workflows. + + +Queues +'''''' + + The patch queue extension integrates quilt functionality into Mercurial. + Changes are maintained as patches which are committed into Mercurial. + Commits can be removed or reordered, and the underlying patch can be + refreshed based on changes made in the working directory. The patch + directory can also be placed under revision control, so you can have a + separate history of changes made to your patches. + + While the mq extension can indeed achieve similar functionality, it's quite + a different workflow than git and may be difficult to adapt to for those + coming from git. + + +Bookmarks +''''''''' + + Bookmarks can be used as an alternative to NamedBranches_ for tracking + multiple lines of development. Systems like Mercurial, CVS, and Subversion + store their branch information as a permanent part of each commit. This + is useful for future auditing of long-lived branches, as it's always + possible to identify which branch a commit was introduced on. Git, by + contrast, has "branches" that are not stored in history, which is useful + for working with numerous short-lived feature branches, but makes future + auditing impossible. Mercurial's bookmark feature is analogous to Git's + branching scheme, but can also be used in conjunction with Mercurial's + traditional named branches. + + Bookmarks are the feature that emulate git workflows most closely and will + therefore be used throughout the remainder of this guide. + + +An introduction to Mercurial bookmarks +-------------------------------------- + +While the functionality may seem entirely analogous to git branches +at first, there are fundamental differences to be aware of: + + +Bookmarks are not git branches +'''''''''''''''''''''''''''''' + +Bookmarks are *not* lightweight Git branches. Bookmarks are simply named +references to commits that are automatically updated when new commits are made. +This, coupled with the creation of new branch heads when committing from a +previous commit provides a workflow similar to git branching, but the fact that +a new branch is *not* created is an important detail to be aware of. This will +be demonstrated in the `Mercurial workflow`_ section. + + +Bookmarks are local +''''''''''''''''''' + +Mercurial bookmarks are intended for local +development and can be deleted with ease, whereas branches cannot. +Note that deleting a bookmark does not mean that the releated changesets +are also deleted. You must use the `strip extension`_ to do that. Bookmarks can +be published to a remote repo to be shared, but must explicitly be pushed (see +http://mercurial.selenic.com/wiki/Bookmarks#Working_with_remote_repositories +for details). + +.. _`strip extension`: http://mercurial.selenic.com/wiki/StripExtension + + + +Mercurial workflow +================== + +The following details hg usage with the above git workflow in mind. The +changeset graphs are displayed using :code:`hg log -G -l [num_commits]`, +which can be tremendously useful when first starting to use hg in order +to help you understand through visuals what it is that hg does with the +branches. + + +Cloning +------- + +Pulling the latest CPython code and looking at the current commits:: + + hg clone https://hg.python.org/cpython + + @ changeset: 93654:bd97eab25c70 + |\ tag: tip + | | parent: 93652:70163e18da87 + | | parent: 93653:21257f916668 + | | user: Ned Deily + | | date: Fri Nov 28 15:22:15 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + | | + | o changeset: 93653:21257f916668 + | | branch: 3.4 + | | parent: 93647:737355f61ba2 + | | user: Ned Deily + | | date: Fri Nov 28 15:21:12 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + | | + o | changeset: 93652:70163e18da87 + | | user: Raymond Hettinger + | | date: Fri Nov 28 14:52:14 2014 -0800 + | | summary: Minor code cleanup. + +*Note that in the above graph, `@` represents your current changeset* + + +Working on issueA +----------------- + +Assuming you've found a bug logged against default tip (master head in git-speak), +create a bookmark, which automatically activates it:: + + hg bookmark issueA + +Now, the history graph should look like this:: + + @ changeset: 93654:bd97eab25c70 + |\ bookmark: issueA + | | tag: tip + | | parent: 93652:70163e18da87 + | | parent: 93653:21257f916668 + | | user: Ned Deily + | | date: Fri Nov 28 15:22:15 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + | | + | o changeset: 93653:21257f916668 + | | branch: 3.4 + | | parent: 93647:737355f61ba2 + | | user: Ned Deily + | | date: Fri Nov 28 15:21:12 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + | | + o | changeset: 93652:70163e18da87 + | | user: Raymond Hettinger + | | date: Fri Nov 28 14:52:14 2014 -0800 + | | summary: Minor code cleanup. + +Notice that the only difference between this and the previous one is that +changeset 91935 now also has the bookmark "issueA". Bookmarks are advanced +automatically with each subsequent commit. + +Once work has been completed on issueA, commit and prepare a patch for +submission to the issue tracker. Note that Mercurial doesn't have git's concept +of staging, so all changes will be committed:: + + hg commit -m 'fix for issueA' + + @ changeset: 93655:a542bc2066d1 + | bookmark: issueA + | tag: tip + | user: Demian Brecht + | date: Thu Dec 04 17:33:42 2014 -0800 + | summary: issueA + | + o changeset: 93654:bd97eab25c70 + |\ parent: 93652:70163e18da87 + | | parent: 93653:21257f916668 + | | user: Ned Deily + | | date: Fri Nov 28 15:22:15 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + | | + o | changeset: 93653:21257f916668 + | | branch: 3.4 + | | parent: 93647:737355f61ba2 + | | user: Ned Deily + | | date: Fri Nov 28 15:21:12 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + +Notice that the new commit's parent was the previous default tip and the +bookmark has automatically been advanced to the new tip. A patch for submission +to the issue tracker can now be prepared with:: + + hg diff -c 93655 > issueA.patch + +The above will diff revision 93655 against its parent. This will work +regardless of the commit that you happen to currently updated to. + + +Working on issueB +----------------- + +Now that the patch has been submitted and it's pending review, work on another +issue can be started. Because of how bookmarks work (and as can be seen in the +previous history graph), the named branch "default" is advanced (remember that +bookmarks are not git branches). A new named branch (such as in git) has not +been created. This means that in order to update the working copy back to the +latest public commit, you must know which commit to revert back to before +creating a new bookmark:: + + hg update 93654 + hg bookmark issueB + + o changeset: 93655:a542bc2066d1 + | bookmark: issueA + | tag: tip + | user: Demian Brecht + | date: Thu Dec 04 17:33:42 2014 -0800 + | summary: issueA + | + @ changeset: 93654:bd97eab25c70 + |\ bookmark: issueB + | | parent: 93652:70163e18da87 + | | parent: 93653:21257f916668 + | | user: Ned Deily + | | date: Fri Nov 28 15:22:15 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + | | + o | changeset: 93653:21257f916668 + | | branch: 3.4 + | | parent: 93647:737355f61ba2 + | | user: Ned Deily + | | date: Fri Nov 28 15:21:12 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + + +Addressing issueA review comments +--------------------------------- + +While working on my new feature, I've received reviews of my bug fix and want to +finish that up before continuing on this much longer feature task. First step is +to commit my current feature work: +While working on issueB, a review has been completed for issueA. The following +demonstrates one method of store current state of issueB, and update back to +issueA:: + + hg commit -m 'issueB WIP' + created new head + +In the above, hg will confirm that a divergent path has been created. This is +intentional and to be expected:: + + hg update issueA + +Reviewing the history graph, a new head can now be seen:: + + o changeset: 93656:6c166f6c1970 + | bookmark: issueB + | tag: tip + | parent: 93654:bd97eab25c70 + | user: Demian Brecht + | date: Thu Dec 04 17:36:36 2014 -0800 + | summary: issueB WIP + | + | @ changeset: 93655:a542bc2066d1 + |/ bookmark: issueA + | user: Demian Brecht + | date: Thu Dec 04 17:33:42 2014 -0800 + | summary: issueA + | + o changeset: 93654:bd97eab25c70 + |\ parent: 93652:70163e18da87 + | | parent: 93653:21257f916668 + | | user: Ned Deily + | | date: Fri Nov 28 15:22:15 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + +Once review comments have been addressed, commit again and prepare an updated +patch. In this case, using :code:`hg commit --amend` will amend the previous +commit with the most recent changes:: + + hg commit --amend + hg diff -c issueA > issueA.patch + + @ changeset: 93656:bba24fde02f0 + | bookmark: issueA + | tag: tip + | parent: 93654:bd97eab25c70 + | user: Demian Brecht + | date: Thu Dec 04 17:33:42 2014 -0800 + | summary: issueA + | + | o changeset: 93655:6c166f6c1970 + |/ bookmark: issueB + | user: Demian Brecht + | date: Thu Dec 04 17:36:36 2014 -0800 + | summary: issueB WIP + | + o changeset: 93654:bd97eab25c70 + |\ parent: 93652:70163e18da87 + | | parent: 93653:21257f916668 + | | user: Ned Deily + | | date: Fri Nov 28 15:22:15 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + + +Continue work on issueB +----------------------- + +Work can now be continued on issueB:: + + hg update issueB + + o changeset: 93656:bba24fde02f0 + | bookmark: issueA + | tag: tip + | parent: 93654:bd97eab25c70 + | user: Demian Brecht + | date: Thu Dec 04 17:33:42 2014 -0800 + | summary: issueA + | + | @ changeset: 93655:6c166f6c1970 + |/ bookmark: issueB + | user: Demian Brecht + | date: Thu Dec 04 17:36:36 2014 -0800 + | summary: issueB WIP + | + o changeset: 93654:bd97eab25c70 + |\ parent: 93652:70163e18da87 + | | parent: 93653:21257f916668 + | | user: Ned Deily + | | date: Fri Nov 28 15:22:15 2014 -0800 + | | summary: Issue #16113: Also remove test_case_sha3_224_huge + + +Rebasing your work +================== + +Rebasing was not previously included as it's an optional step. + +As patches sometimes take time to have merged, there can be times when you'll +need to re-apply commits against the latest version in the public repo. Using +git, you might do this:: + + git pull --rebase + +The Mercurial equivalent is:: + + hg pull --rebase + +As this alters history (which Mercurial largely avoids in practice), the rebase +extension will need to be enabled. To enable the rebase extension, it must be +added to your .hgrc file:: + + [extensions] + rebase = + + +Workflow comparison +=================== + +Comparing against the git workflow above (skipping optional steps), the hg +equivalent in its entirety looks like this:: + + # git clone git at github.com:python/cpython.git + hg clone https://hg.python.org/cpython + + # work on issueA + # git checkout -b issueA + # git commit -a + hg bookmark issueA + hg commit + + # start work on issueB + # git checkout master + hg update [revision_number] + + # git checkout -b issueB + # git commit -a + hg bookmark issueB + hg commit + + # address review comments + # git checkout issueA + # git commit -a + hg update issueA + hg commit --amend + + # create patch + # git diff master..issueA > issueA.patch + hg diff -c issueA > issueA.patch + + # continue working on issueB + # git checkout issueB + hg update issueB + + +.. _NamedBranches: http://mercurial.selenic.com/wiki/NamedBranches diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -49,6 +49,7 @@ * `Buildbot status`_ * :doc:`faq` * PEPs_ (Python Enhancement Proposals) +* :doc:`gitdevs` .. _contributing: @@ -83,6 +84,7 @@ * :doc:`devcycle` * :doc:`buildbots` * :doc:`coverity` +* :doc:`gitdevs` It is **recommended** that the above documents be read in the order listed. You can stop where you feel comfortable and begin contributing immediately without @@ -209,6 +211,7 @@ compiler coverity clang + gitdevs faq -- Repository URL: https://hg.python.org/devguide From python-checkins at python.org Sat Jan 17 02:46:56 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 17 Jan 2015 01:46:56 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_capitialize_?= =?utf-8?q?=22HttpOnly=22_and_=22Secure=22_as_they_appear_in_the_standard_?= =?utf-8?q?and_other?= Message-ID: <20150117014650.103307.2436@psf.io> https://hg.python.org/cpython/rev/0d8380c493ad changeset: 94188:0d8380c493ad branch: 3.4 parent: 94186:992ce0dcfb29 user: Benjamin Peterson date: Fri Jan 16 20:43:55 2015 -0500 summary: capitialize "HttpOnly" and "Secure" as they appear in the standard and other impls (closes #23250) Patch by Jon Dufresne. files: Lib/http/cookies.py | 4 ++-- Lib/test/test_http_cookies.py | 4 ++-- Misc/NEWS | 3 +++ 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Lib/http/cookies.py b/Lib/http/cookies.py --- a/Lib/http/cookies.py +++ b/Lib/http/cookies.py @@ -330,8 +330,8 @@ "comment" : "Comment", "domain" : "Domain", "max-age" : "Max-Age", - "secure" : "secure", - "httponly" : "httponly", + "secure" : "Secure", + "httponly" : "HttpOnly", "version" : "Version", } diff --git a/Lib/test/test_http_cookies.py b/Lib/test/test_http_cookies.py --- a/Lib/test/test_http_cookies.py +++ b/Lib/test/test_http_cookies.py @@ -114,7 +114,7 @@ C['Customer']['secure'] = True C['Customer']['httponly'] = True self.assertEqual(C.output(), - 'Set-Cookie: Customer="WILE_E_COYOTE"; httponly; secure') + 'Set-Cookie: Customer="WILE_E_COYOTE"; HttpOnly; Secure') def test_secure_httponly_false_if_not_present(self): C = cookies.SimpleCookie() @@ -152,7 +152,7 @@ C = cookies.SimpleCookie() C.load('eggs = scrambled ; secure ; path = bar ; foo=foo ') self.assertEqual(C.output(), - 'Set-Cookie: eggs=scrambled; Path=bar; secure\r\nSet-Cookie: foo=foo') + 'Set-Cookie: eggs=scrambled; Path=bar; Secure\r\nSet-Cookie: foo=foo') def test_quoted_meta(self): # Try cookie with quoted meta-data diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #23250: In the http.cookies module, capitalize "HttpOnly" and "Secure" + as they are written in the standard. + - Issue #23063: In the disutils' check command, fix parsing of reST with code or code-block directives. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 17 02:46:56 2015 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 17 Jan 2015 01:46:56 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjI5ODYp?= Message-ID: <20150117014650.103287.41894@psf.io> https://hg.python.org/cpython/rev/d3671e6ba106 changeset: 94189:d3671e6ba106 parent: 94187:031fc0231f3d parent: 94188:0d8380c493ad user: Benjamin Peterson date: Fri Jan 16 20:46:37 2015 -0500 summary: merge 3.4 (#22986) files: Lib/http/cookies.py | 4 ++-- Lib/test/test_http_cookies.py | 4 ++-- Misc/NEWS | 3 +++ 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Lib/http/cookies.py b/Lib/http/cookies.py --- a/Lib/http/cookies.py +++ b/Lib/http/cookies.py @@ -330,8 +330,8 @@ "comment" : "Comment", "domain" : "Domain", "max-age" : "Max-Age", - "secure" : "secure", - "httponly" : "httponly", + "secure" : "Secure", + "httponly" : "HttpOnly", "version" : "Version", } diff --git a/Lib/test/test_http_cookies.py b/Lib/test/test_http_cookies.py --- a/Lib/test/test_http_cookies.py +++ b/Lib/test/test_http_cookies.py @@ -114,7 +114,7 @@ C['Customer']['secure'] = True C['Customer']['httponly'] = True self.assertEqual(C.output(), - 'Set-Cookie: Customer="WILE_E_COYOTE"; httponly; secure') + 'Set-Cookie: Customer="WILE_E_COYOTE"; HttpOnly; Secure') def test_secure_httponly_false_if_not_present(self): C = cookies.SimpleCookie() @@ -145,7 +145,7 @@ C = cookies.SimpleCookie() C.load('eggs = scrambled ; secure ; path = bar ; foo=foo ') self.assertEqual(C.output(), - 'Set-Cookie: eggs=scrambled; Path=bar; secure\r\nSet-Cookie: foo=foo') + 'Set-Cookie: eggs=scrambled; Path=bar; Secure\r\nSet-Cookie: foo=foo') def test_quoted_meta(self): # Try cookie with quoted meta-data diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,9 @@ Library ------- +- Issue #23250: In the http.cookies module, capitalize "HttpOnly" and "Secure" + as they are written in the standard. + - Issue #23063: In the disutils' check command, fix parsing of reST with code or code-block directives. -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Sat Jan 17 09:12:22 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 17 Jan 2015 09:12:22 +0100 Subject: [Python-checkins] Daily reference leaks (d3671e6ba106): sum=6 Message-ID: results for d3671e6ba106 on branch "default" -------------------------------------------- test_asyncio leaked [0, 3, 0] memory blocks, sum=3 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogPygmlD', '-x'] From python-checkins at python.org Sat Jan 17 15:53:19 2015 From: python-checkins at python.org (zach.ware) Date: Sat, 17 Jan 2015 14:53:19 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMzI1?= =?utf-8?q?6=3A_Avoid_a_crash_in_test=5Fctypes?= Message-ID: <20150117145318.118086.59754@psf.io> https://hg.python.org/cpython/rev/2bbd7b739b85 changeset: 94190:2bbd7b739b85 branch: 3.4 parent: 94188:0d8380c493ad user: Zachary Ware date: Sat Jan 17 08:50:42 2015 -0600 summary: Closes #23256: Avoid a crash in test_ctypes Only happened with oddly capitalized debug executables on Windows. Patch by Claudiu Popa. files: Lib/ctypes/test/test_win32.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ctypes/test/test_win32.py b/Lib/ctypes/test/test_win32.py --- a/Lib/ctypes/test/test_win32.py +++ b/Lib/ctypes/test/test_win32.py @@ -38,7 +38,7 @@ @unittest.skipUnless(sys.platform == "win32", 'Windows-specific test') class FunctionCallTestCase(unittest.TestCase): @unittest.skipUnless('MSC' in sys.version, "SEH only supported by MSC") - @unittest.skipIf(sys.executable.endswith('_d.exe'), + @unittest.skipIf(sys.executable.lower().endswith('_d.exe'), "SEH not enabled in debug builds") def test_SEH(self): # Call functions with invalid arguments, and make sure -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 17 15:53:19 2015 From: python-checkins at python.org (zach.ware) Date: Sat, 17 Jan 2015 14:53:19 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2Ugd2l0aCAzLjQgKCMyMzI1Nik=?= Message-ID: <20150117145319.103293.70409@psf.io> https://hg.python.org/cpython/rev/b5821f7493c1 changeset: 94191:b5821f7493c1 parent: 94189:d3671e6ba106 parent: 94190:2bbd7b739b85 user: Zachary Ware date: Sat Jan 17 08:53:09 2015 -0600 summary: Merge with 3.4 (#23256) files: Lib/ctypes/test/test_win32.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ctypes/test/test_win32.py b/Lib/ctypes/test/test_win32.py --- a/Lib/ctypes/test/test_win32.py +++ b/Lib/ctypes/test/test_win32.py @@ -38,7 +38,7 @@ @unittest.skipUnless(sys.platform == "win32", 'Windows-specific test') class FunctionCallTestCase(unittest.TestCase): @unittest.skipUnless('MSC' in sys.version, "SEH only supported by MSC") - @unittest.skipIf(sys.executable.endswith('_d.exe'), + @unittest.skipIf(sys.executable.lower().endswith('_d.exe'), "SEH not enabled in debug builds") def test_SEH(self): # Call functions with invalid arguments, and make sure -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 17 16:22:29 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 17 Jan 2015 15:22:29 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315955=3A_Add_an_o?= =?utf-8?q?ption_to_limit_output_size_when_decompressing_LZMA_data=2E?= Message-ID: <20150117152228.118092.91445@psf.io> https://hg.python.org/cpython/rev/00e552a23bcc changeset: 94192:00e552a23bcc user: Antoine Pitrou date: Sat Jan 17 16:22:18 2015 +0100 summary: Issue #15955: Add an option to limit output size when decompressing LZMA data. Patch by Nikolaus Rath and Martin Panter. files: Doc/library/lzma.rst | 37 ++- Lib/test/test_lzma.py | 91 ++++++++ Misc/NEWS | 3 + Modules/_lzmamodule.c | 218 +++++++++++++++++--- Modules/clinic/_lzmamodule.c.h | 37 ++- 5 files changed, 327 insertions(+), 59 deletions(-) diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -221,13 +221,32 @@ decompress a multi-stream input with :class:`LZMADecompressor`, you must create a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) - Decompress *data* (a :class:`bytes` object), returning a :class:`bytes` - object containing the decompressed data for at least part of the input. - Some of *data* may be buffered internally, for use in later calls to - :meth:`decompress`. The returned data should be concatenated with the - output of any previous calls to :meth:`decompress`. + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. + + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. + + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: check @@ -245,6 +264,12 @@ Before the end of the stream is reached, this will be ``b""``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 .. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None) diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -135,6 +135,97 @@ self.assertTrue(lzd.eof) self.assertEqual(lzd.unused_data, b"") + def test_decompressor_chunks_maxsize(self): + lzd = LZMADecompressor() + max_length = 100 + out = [] + + # Feed first half the input + len_ = len(COMPRESSED_XZ) // 2 + out.append(lzd.decompress(COMPRESSED_XZ[:len_], + max_length=max_length)) + self.assertFalse(lzd.needs_input) + self.assertEqual(len(out[-1]), max_length) + + # Retrieve more data without providing more input + out.append(lzd.decompress(b'', max_length=max_length)) + self.assertFalse(lzd.needs_input) + self.assertEqual(len(out[-1]), max_length) + + # Retrieve more data while providing more input + out.append(lzd.decompress(COMPRESSED_XZ[len_:], + max_length=max_length)) + self.assertLessEqual(len(out[-1]), max_length) + + # Retrieve remaining uncompressed data + while not lzd.eof: + out.append(lzd.decompress(b'', max_length=max_length)) + self.assertLessEqual(len(out[-1]), max_length) + + out = b"".join(out) + self.assertEqual(out, INPUT) + self.assertEqual(lzd.check, lzma.CHECK_CRC64) + self.assertEqual(lzd.unused_data, b"") + + def test_decompressor_inputbuf_1(self): + # Test reusing input buffer after moving existing + # contents to beginning + lzd = LZMADecompressor() + out = [] + + # Create input buffer and fill it + self.assertEqual(lzd.decompress(COMPRESSED_XZ[:100], + max_length=0), b'') + + # Retrieve some results, freeing capacity at beginning + # of input buffer + out.append(lzd.decompress(b'', 2)) + + # Add more data that fits into input buffer after + # moving existing data to beginning + out.append(lzd.decompress(COMPRESSED_XZ[100:105], 15)) + + # Decompress rest of data + out.append(lzd.decompress(COMPRESSED_XZ[105:])) + self.assertEqual(b''.join(out), INPUT) + + def test_decompressor_inputbuf_2(self): + # Test reusing input buffer by appending data at the + # end right away + lzd = LZMADecompressor() + out = [] + + # Create input buffer and empty it + self.assertEqual(lzd.decompress(COMPRESSED_XZ[:200], + max_length=0), b'') + out.append(lzd.decompress(b'')) + + # Fill buffer with new data + out.append(lzd.decompress(COMPRESSED_XZ[200:280], 2)) + + # Append some more data, not enough to require resize + out.append(lzd.decompress(COMPRESSED_XZ[280:300], 2)) + + # Decompress rest of data + out.append(lzd.decompress(COMPRESSED_XZ[300:])) + self.assertEqual(b''.join(out), INPUT) + + def test_decompressor_inputbuf_3(self): + # Test reusing input buffer after extending it + + lzd = LZMADecompressor() + out = [] + + # Create almost full input buffer + out.append(lzd.decompress(COMPRESSED_XZ[:200], 5)) + + # Add even more data to it, requiring resize + out.append(lzd.decompress(COMPRESSED_XZ[200:300], 5)) + + # Decompress rest of data + out.append(lzd.decompress(COMPRESSED_XZ[300:])) + self.assertEqual(b''.join(out), INPUT) + def test_decompressor_unused_data(self): lzd = LZMADecompressor() extra = b"fooblibar" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,9 @@ Library ------- +- Issue #15955: Add an option to limit output size when decompressing LZMA + data. Patch by Nikolaus Rath and Martin Panter. + - Issue #23250: In the http.cookies module, capitalize "HttpOnly" and "Secure" as they are written in the standard. diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -66,6 +66,9 @@ int check; char eof; PyObject *unused_data; + char needs_input; + uint8_t *input_buffer; + size_t input_buffer_size; #ifdef WITH_THREAD PyThread_type_lock lock; #endif @@ -142,10 +145,15 @@ #endif static int -grow_buffer(PyObject **buf) +grow_buffer(PyObject **buf, Py_ssize_t max_length) { - size_t size = PyBytes_GET_SIZE(*buf); - return _PyBytes_Resize(buf, size + (size >> 3) + 6); + Py_ssize_t size = PyBytes_GET_SIZE(*buf); + Py_ssize_t newsize = size + (size >> 3) + 6; + + if (max_length > 0 && newsize > max_length) + newsize = max_length; + + return _PyBytes_Resize(buf, newsize); } @@ -504,7 +512,7 @@ static PyObject * compress(Compressor *c, uint8_t *data, size_t len, lzma_action action) { - size_t data_size = 0; + Py_ssize_t data_size = 0; PyObject *result; result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE); @@ -527,13 +535,13 @@ (action == LZMA_FINISH && lzret == LZMA_STREAM_END)) { break; } else if (c->lzs.avail_out == 0) { - if (grow_buffer(&result) == -1) + if (grow_buffer(&result, -1) == -1) goto error; c->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result) + data_size; c->lzs.avail_out = PyBytes_GET_SIZE(result) - data_size; } } - if (data_size != (size_t)PyBytes_GET_SIZE(result)) + if (data_size != PyBytes_GET_SIZE(result)) if (_PyBytes_Resize(&result, data_size) == -1) goto error; return result; @@ -888,25 +896,33 @@ /* LZMADecompressor class. */ -static PyObject * -decompress(Decompressor *d, uint8_t *data, size_t len) +/* Decompress data of length d->lzs.avail_in in d->lzs.next_in. The output + buffer is allocated dynamically and returned. At most max_length bytes are + returned, so some of the input may not be consumed. d->lzs.next_in and + d->lzs.avail_in are updated to reflect the consumed input. */ +static PyObject* +decompress_buf(Decompressor *d, Py_ssize_t max_length) { - size_t data_size = 0; + Py_ssize_t data_size = 0; PyObject *result; - - result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE); + lzma_stream *lzs = &d->lzs; + + if (max_length < 0 || max_length >= INITIAL_BUFFER_SIZE) + result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE); + else + result = PyBytes_FromStringAndSize(NULL, max_length); if (result == NULL) return NULL; - d->lzs.next_in = data; - d->lzs.avail_in = len; - d->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result); - d->lzs.avail_out = PyBytes_GET_SIZE(result); + + lzs->next_out = (uint8_t *)PyBytes_AS_STRING(result); + lzs->avail_out = PyBytes_GET_SIZE(result); + for (;;) { lzma_ret lzret; Py_BEGIN_ALLOW_THREADS - lzret = lzma_code(&d->lzs, LZMA_RUN); - data_size = (char *)d->lzs.next_out - PyBytes_AS_STRING(result); + lzret = lzma_code(lzs, LZMA_RUN); + data_size = (char *)lzs->next_out - PyBytes_AS_STRING(result); Py_END_ALLOW_THREADS if (catch_lzma_error(lzret)) goto error; @@ -914,26 +930,131 @@ d->check = lzma_get_check(&d->lzs); if (lzret == LZMA_STREAM_END) { d->eof = 1; - if (d->lzs.avail_in > 0) { - Py_CLEAR(d->unused_data); - d->unused_data = PyBytes_FromStringAndSize( - (char *)d->lzs.next_in, d->lzs.avail_in); - if (d->unused_data == NULL) - goto error; - } break; - } else if (d->lzs.avail_in == 0) { + } else if (lzs->avail_in == 0) { break; - } else if (d->lzs.avail_out == 0) { - if (grow_buffer(&result) == -1) + } else if (lzs->avail_out == 0) { + if (data_size == max_length) + break; + if (grow_buffer(&result, max_length) == -1) goto error; - d->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result) + data_size; - d->lzs.avail_out = PyBytes_GET_SIZE(result) - data_size; + lzs->next_out = (uint8_t *)PyBytes_AS_STRING(result) + data_size; + lzs->avail_out = PyBytes_GET_SIZE(result) - data_size; } } - if (data_size != (size_t)PyBytes_GET_SIZE(result)) + if (data_size != PyBytes_GET_SIZE(result)) if (_PyBytes_Resize(&result, data_size) == -1) goto error; + + return result; + +error: + Py_XDECREF(result); + return NULL; +} + +static PyObject * +decompress(Decompressor *d, uint8_t *data, size_t len, Py_ssize_t max_length) +{ + char input_buffer_in_use; + PyObject *result; + lzma_stream *lzs = &d->lzs; + + /* Prepend unconsumed input if necessary */ + if (lzs->next_in != NULL) { + size_t avail_now, avail_total; + + /* Number of bytes we can append to input buffer */ + avail_now = (d->input_buffer + d->input_buffer_size) + - (lzs->next_in + lzs->avail_in); + + /* Number of bytes we can append if we move existing + contents to beginning of buffer (overwriting + consumed input) */ + avail_total = d->input_buffer_size - lzs->avail_in; + + if (avail_total < len) { + size_t offset = lzs->next_in - d->input_buffer; + uint8_t *tmp; + size_t new_size = d->input_buffer_size + len - avail_now; + + /* Assign to temporary variable first, so we don't + lose address of allocated buffer if realloc fails */ + tmp = PyMem_Realloc(d->input_buffer, new_size); + if (tmp == NULL) { + PyErr_SetNone(PyExc_MemoryError); + return NULL; + } + d->input_buffer = tmp; + d->input_buffer_size = new_size; + + lzs->next_in = d->input_buffer + offset; + } + else if (avail_now < len) { + memmove(d->input_buffer, lzs->next_in, + lzs->avail_in); + lzs->next_in = d->input_buffer; + } + memcpy((void*)(lzs->next_in + lzs->avail_in), data, len); + lzs->avail_in += len; + input_buffer_in_use = 1; + } + else { + lzs->next_in = data; + lzs->avail_in = len; + input_buffer_in_use = 0; + } + + result = decompress_buf(d, max_length); + if(result == NULL) + return NULL; + + if (d->eof) { + d->needs_input = 0; + if (lzs->avail_in > 0) { + Py_CLEAR(d->unused_data); + d->unused_data = PyBytes_FromStringAndSize( + (char *)lzs->next_in, lzs->avail_in); + if (d->unused_data == NULL) + goto error; + } + } + else if (lzs->avail_in == 0) { + lzs->next_in = NULL; + d->needs_input = 1; + } + else { + d->needs_input = 0; + + /* If we did not use the input buffer, we now have + to copy the tail from the caller's buffer into the + input buffer */ + if (!input_buffer_in_use) { + + /* Discard buffer if it's too small + (resizing it may needlessly copy the current contents) */ + if (d->input_buffer != NULL && + d->input_buffer_size < lzs->avail_in) { + PyMem_Free(d->input_buffer); + d->input_buffer = NULL; + } + + /* Allocate if necessary */ + if (d->input_buffer == NULL) { + d->input_buffer = PyMem_Malloc(lzs->avail_in); + if (d->input_buffer == NULL) { + PyErr_SetNone(PyExc_MemoryError); + goto error; + } + d->input_buffer_size = lzs->avail_in; + } + + /* Copy tail */ + memcpy(d->input_buffer, lzs->next_in, lzs->avail_in); + lzs->next_in = d->input_buffer; + } + } + return result; error: @@ -946,20 +1067,27 @@ self: self(type="Decompressor *") data: Py_buffer - / + max_length: Py_ssize_t=-1 -Provide data to the decompressor object. +Decompress *data*, returning uncompressed data as bytes. -Returns a chunk of decompressed data if possible, or b'' otherwise. +If *max_length* is nonnegative, returns at most *max_length* bytes of +decompressed data. If this limit is reached and further output can be +produced, *self.needs_input* will be set to ``False``. In this case, the next +call to *decompress()* may provide *data* as b'' to obtain more of the output. -Attempting to decompress data after the end of stream is reached -raises an EOFError. Any data found after the end of the stream -is ignored and saved in the unused_data attribute. +If all of the input data was decompressed and returned (either because this +was less than *max_length* bytes, or because *max_length* was negative), +*self.needs_input* will be set to True. + +Attempting to decompress data after the end of stream is reached raises an +EOFError. Any data found after the end of the stream is ignored and saved in +the unused_data attribute. [clinic start generated code]*/ static PyObject * -_lzma_LZMADecompressor_decompress_impl(Decompressor *self, Py_buffer *data) -/*[clinic end generated code: output=d86e78da7ff0ff21 input=50c4768b821bf0ef]*/ +_lzma_LZMADecompressor_decompress_impl(Decompressor *self, Py_buffer *data, Py_ssize_t max_length) +/*[clinic end generated code: output=1532a5bb23629001 input=262e4e217f49039b]*/ { PyObject *result = NULL; @@ -967,7 +1095,7 @@ if (self->eof) PyErr_SetString(PyExc_EOFError, "Already at end of stream"); else - result = decompress(self, data->buf, data->len); + result = decompress(self, data->buf, data->len, max_length); RELEASE_LOCK(self); return result; } @@ -1055,6 +1183,7 @@ self->alloc.alloc = PyLzma_Malloc; self->alloc.free = PyLzma_Free; self->lzs.allocator = &self->alloc; + self->lzs.next_in = NULL; #ifdef WITH_THREAD self->lock = PyThread_allocate_lock(); @@ -1065,6 +1194,9 @@ #endif self->check = LZMA_CHECK_UNKNOWN; + self->needs_input = 1; + self->input_buffer = NULL; + self->input_buffer_size = 0; self->unused_data = PyBytes_FromStringAndSize(NULL, 0); if (self->unused_data == NULL) goto error; @@ -1113,6 +1245,9 @@ static void Decompressor_dealloc(Decompressor *self) { + if(self->input_buffer != NULL) + PyMem_Free(self->input_buffer); + lzma_end(&self->lzs); Py_CLEAR(self->unused_data); #ifdef WITH_THREAD @@ -1134,6 +1269,9 @@ PyDoc_STRVAR(Decompressor_eof_doc, "True if the end-of-stream marker has been reached."); +PyDoc_STRVAR(Decompressor_needs_input_doc, +"True if more input is needed before more decompressed data can be produced."); + PyDoc_STRVAR(Decompressor_unused_data_doc, "Data found after the end of the compressed stream."); @@ -1142,6 +1280,8 @@ Decompressor_check_doc}, {"eof", T_BOOL, offsetof(Decompressor, eof), READONLY, Decompressor_eof_doc}, + {"needs_input", T_BOOL, offsetof(Decompressor, needs_input), READONLY, + Decompressor_needs_input_doc}, {"unused_data", T_OBJECT_EX, offsetof(Decompressor, unused_data), READONLY, Decompressor_unused_data_doc}, {NULL} diff --git a/Modules/clinic/_lzmamodule.c.h b/Modules/clinic/_lzmamodule.c.h --- a/Modules/clinic/_lzmamodule.c.h +++ b/Modules/clinic/_lzmamodule.c.h @@ -62,34 +62,43 @@ } PyDoc_STRVAR(_lzma_LZMADecompressor_decompress__doc__, -"decompress($self, data, /)\n" +"decompress($self, /, data, max_length=-1)\n" "--\n" "\n" -"Provide data to the decompressor object.\n" +"Decompresses *data*, returning uncompressed data as bytes.\n" "\n" -"Returns a chunk of decompressed data if possible, or b\'\' otherwise.\n" +"If *max_length* is nonnegative, returns at most *max_length* bytes of\n" +"decompressed data. If this limit is reached and further output can be\n" +"produced, *self.needs_input* will be set to ``False``. In this case, the next\n" +"call to *decompress()* may provide *data* as b\'\' to obtain more of the output.\n" "\n" -"Attempting to decompress data after the end of stream is reached\n" -"raises an EOFError. Any data found after the end of the stream\n" -"is ignored and saved in the unused_data attribute."); +"If all of the input data was decompressed and returned (either because this\n" +"was less than *max_length* bytes, or because *max_length* was negative),\n" +"*self.needs_input* will be set to True.\n" +"\n" +"Attempting to decompress data after the end of stream is reached raises an\n" +"EOFError. Any data found after the end of the stream is ignored and saved in\n" +"the unused_data attribute."); #define _LZMA_LZMADECOMPRESSOR_DECOMPRESS_METHODDEF \ - {"decompress", (PyCFunction)_lzma_LZMADecompressor_decompress, METH_VARARGS, _lzma_LZMADecompressor_decompress__doc__}, + {"decompress", (PyCFunction)_lzma_LZMADecompressor_decompress, METH_VARARGS|METH_KEYWORDS, _lzma_LZMADecompressor_decompress__doc__}, static PyObject * -_lzma_LZMADecompressor_decompress_impl(Decompressor *self, Py_buffer *data); +_lzma_LZMADecompressor_decompress_impl(Decompressor *self, Py_buffer *data, Py_ssize_t max_length); static PyObject * -_lzma_LZMADecompressor_decompress(Decompressor *self, PyObject *args) +_lzma_LZMADecompressor_decompress(Decompressor *self, PyObject *args, PyObject *kwargs) { PyObject *return_value = NULL; + static char *_keywords[] = {"data", "max_length", NULL}; Py_buffer data = {NULL, NULL}; + Py_ssize_t max_length = -1; - if (!PyArg_ParseTuple(args, - "y*:decompress", - &data)) + if (!PyArg_ParseTupleAndKeywords(args, kwargs, + "y*|n:decompress", _keywords, + &data, &max_length)) goto exit; - return_value = _lzma_LZMADecompressor_decompress_impl(self, &data); + return_value = _lzma_LZMADecompressor_decompress_impl(self, &data, max_length); exit: /* Cleanup for data */ @@ -242,4 +251,4 @@ return return_value; } -/*[clinic end generated code: output=808fec8216ac712b input=a9049054013a1b77]*/ +/*[clinic end generated code: output=d17fac38b09626d8 input=a9049054013a1b77]*/ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 17 20:02:48 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 17 Jan 2015 19:02:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321817=3A_When_an_?= =?utf-8?q?exception_is_raised_in_a_task_submitted_to_a?= Message-ID: <20150117190246.93183.56507@psf.io> https://hg.python.org/cpython/rev/a36b402b099b changeset: 94193:a36b402b099b user: Antoine Pitrou date: Sat Jan 17 20:02:14 2015 +0100 summary: Issue #21817: When an exception is raised in a task submitted to a ProcessPoolExecutor, the remote traceback is now displayed in the parent process. Patch by Claudiu Popa. files: Lib/concurrent/futures/process.py | 26 +++++++++++++++- Lib/test/test_concurrent_futures.py | 26 +++++++++++++++++ Misc/NEWS | 4 ++ 3 files changed, 54 insertions(+), 2 deletions(-) diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py --- a/Lib/concurrent/futures/process.py +++ b/Lib/concurrent/futures/process.py @@ -57,6 +57,7 @@ import weakref from functools import partial import itertools +import traceback # Workers are created as daemon threads and processes. This is done to allow the # interpreter to exit when there are still idle processes in a @@ -90,6 +91,27 @@ # (Futures in the call queue cannot be cancelled). EXTRA_QUEUED_CALLS = 1 +# Hack to embed stringification of remote traceback in local traceback + +class _RemoteTraceback(Exception): + def __init__(self, tb): + self.tb = tb + def __str__(self): + return self.tb + +class _ExceptionWithTraceback: + def __init__(self, exc, tb): + tb = traceback.format_exception(type(exc), exc, tb) + tb = ''.join(tb) + self.exc = exc + self.tb = '\n"""\n%s"""' % tb + def __reduce__(self): + return _rebuild_exc, (self.exc, self.tb) + +def _rebuild_exc(exc, tb): + exc.__cause__ = _RemoteTraceback(tb) + return exc + class _WorkItem(object): def __init__(self, future, fn, args, kwargs): self.future = future @@ -152,8 +174,8 @@ try: r = call_item.fn(*call_item.args, **call_item.kwargs) except BaseException as e: - result_queue.put(_ResultItem(call_item.work_id, - exception=e)) + exc = _ExceptionWithTraceback(e, e.__traceback__) + result_queue.put(_ResultItem(call_item.work_id, exception=exc)) else: result_queue.put(_ResultItem(call_item.work_id, result=r)) diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -480,6 +480,32 @@ ref) self.assertRaises(ValueError, bad_map) + @classmethod + def _test_traceback(cls): + raise RuntimeError(123) # some comment + + def test_traceback(self): + # We want ensure that the traceback from the child process is + # contained in the traceback raised in the main process. + future = self.executor.submit(self._test_traceback) + with self.assertRaises(Exception) as cm: + future.result() + + exc = cm.exception + self.assertIs(type(exc), RuntimeError) + self.assertEqual(exc.args, (123,)) + cause = exc.__cause__ + self.assertIs(type(cause), futures.process._RemoteTraceback) + self.assertIn('raise RuntimeError(123) # some comment', cause.tb) + + with test.support.captured_stderr() as f1: + try: + raise exc + except RuntimeError: + sys.excepthook(*sys.exc_info()) + self.assertIn('raise RuntimeError(123) # some comment', + f1.getvalue()) + class FutureTests(unittest.TestCase): def test_done_callback_with_result(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,10 @@ Library ------- +- Issue #21817: When an exception is raised in a task submitted to a + ProcessPoolExecutor, the remote traceback is now displayed in the + parent process. Patch by Claudiu Popa. + - Issue #15955: Add an option to limit output size when decompressing LZMA data. Patch by Nikolaus Rath and Martin Panter. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 17 23:40:28 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 17 Jan 2015 22:40:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Updated_copyright_year_=28?= =?utf-8?q?after_my_patch_from_last_year_reverted_it=29?= Message-ID: <20150117222940.93181.86275@psf.io> https://hg.python.org/cpython/rev/47c850e0539b changeset: 94194:47c850e0539b user: Steve Dower date: Sat Jan 17 14:29:01 2015 -0800 summary: Updated copyright year (after my patch from last year reverted it) files: PC/python_ver_rc.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/PC/python_ver_rc.h b/PC/python_ver_rc.h --- a/PC/python_ver_rc.h +++ b/PC/python_ver_rc.h @@ -4,7 +4,7 @@ #include "winver.h" #define PYTHON_COMPANY "Python Software Foundation" -#define PYTHON_COPYRIGHT "Copyright ? 2001-2014 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC." +#define PYTHON_COPYRIGHT "Copyright ? 2001-2015 Python Software Foundation. Copyright ? 2000 BeOpen.com. Copyright ? 1995-2001 CNRI. Copyright ? 1991-1995 SMC." #define MS_WINDOWS #include "modsupport.h" -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 02:00:17 2015 From: python-checkins at python.org (ned.deily) Date: Sun, 18 Jan 2015 01:00:17 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjEx?= =?utf-8?q?=3A_Workaround_test=5Flogging_failure_on_some_OS_X_10=2E6_syste?= =?utf-8?b?bXM6?= Message-ID: <20150118010015.69924.49709@psf.io> https://hg.python.org/cpython/rev/90b664532d1c changeset: 94195:90b664532d1c branch: 3.4 parent: 94190:2bbd7b739b85 user: Ned Deily date: Sat Jan 17 16:57:19 2015 -0800 summary: Issue #23211: Workaround test_logging failure on some OS X 10.6 systems: getaddrinfo("localhost") can fail depending on the name server configuration, use "127.0.0.0" instead. files: Lib/test/test_logging.py | 4 ++-- Misc/NEWS | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -930,10 +930,10 @@ TIMEOUT = 8.0 def test_basic(self): sockmap = {} - server = TestSMTPServer(('localhost', 0), self.process_message, 0.001, + server = TestSMTPServer((support.HOST, 0), self.process_message, 0.001, sockmap) server.start() - addr = ('localhost', server.port) + addr = (support.HOST, server.port) h = logging.handlers.SMTPHandler(addr, 'me', 'you', 'Log', timeout=self.TIMEOUT) self.assertEqual(h.toaddrs, ['you']) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -290,6 +290,8 @@ - Issue #22770: Prevent some Tk segfaults on OS X when running gui tests. +- Issue #23211: Workaround test_logging failure on some OS X 10.6 systems. + Build ----- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 02:00:17 2015 From: python-checkins at python.org (ned.deily) Date: Sun, 18 Jan 2015 01:00:17 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323211=3A_merge_from_3=2E4?= Message-ID: <20150118010016.103313.65149@psf.io> https://hg.python.org/cpython/rev/e3dfe942697e changeset: 94196:e3dfe942697e parent: 94194:47c850e0539b parent: 94195:90b664532d1c user: Ned Deily date: Sat Jan 17 16:59:50 2015 -0800 summary: Issue #23211: merge from 3.4 files: Lib/test/test_logging.py | 4 ++-- Misc/NEWS | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -926,10 +926,10 @@ TIMEOUT = 8.0 def test_basic(self): sockmap = {} - server = TestSMTPServer(('localhost', 0), self.process_message, 0.001, + server = TestSMTPServer((support.HOST, 0), self.process_message, 0.001, sockmap) server.start() - addr = ('localhost', server.port) + addr = (support.HOST, server.port) h = logging.handlers.SMTPHandler(addr, 'me', 'you', 'Log', timeout=self.TIMEOUT) self.assertEqual(h.toaddrs, ['you']) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1611,6 +1611,8 @@ - Issue #22770: Prevent some Tk segfaults on OS X when running gui tests. +- Issue #23211: Workaround test_logging failure on some OS X 10.6 systems. + Tools/Demos ----------- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 02:34:15 2015 From: python-checkins at python.org (ned.deily) Date: Sun, 18 Jan 2015 01:34:15 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323211=3A_null_merge?= Message-ID: <20150118013413.103291.82343@psf.io> https://hg.python.org/cpython/rev/2d71d0f954fb changeset: 94198:2d71d0f954fb parent: 94196:e3dfe942697e parent: 94197:65ac2b992673 user: Ned Deily date: Sat Jan 17 17:33:49 2015 -0800 summary: Issue #23211: null merge files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 02:34:15 2015 From: python-checkins at python.org (ned.deily) Date: Sun, 18 Jan 2015 01:34:15 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjEx?= =?utf-8?q?=3A_Fix_patch_for_3=2E4_differences=2E?= Message-ID: <20150118013412.104124.2460@psf.io> https://hg.python.org/cpython/rev/65ac2b992673 changeset: 94197:65ac2b992673 branch: 3.4 parent: 94195:90b664532d1c user: Ned Deily date: Sat Jan 17 17:31:13 2015 -0800 summary: Issue #23211: Fix patch for 3.4 differences. files: Lib/test/test_logging.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -41,7 +41,7 @@ import tempfile from test.script_helper import assert_python_ok from test.support import (captured_stdout, run_with_locale, run_unittest, - patch, requires_zlib, TestHandler, Matcher) + patch, requires_zlib, TestHandler, Matcher, HOST) import textwrap import time import unittest @@ -930,10 +930,10 @@ TIMEOUT = 8.0 def test_basic(self): sockmap = {} - server = TestSMTPServer((support.HOST, 0), self.process_message, 0.001, + server = TestSMTPServer((HOST, 0), self.process_message, 0.001, sockmap) server.start() - addr = (support.HOST, server.port) + addr = (HOST, server.port) h = logging.handlers.SMTPHandler(addr, 'me', 'you', 'Log', timeout=self.TIMEOUT) self.assertEqual(h.toaddrs, ['you']) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 06:10:34 2015 From: python-checkins at python.org (ned.deily) Date: Sun, 18 Jan 2015 05:10:34 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTgw?= =?utf-8?q?=3A_Rename_IDLE_=22Windows=22_menu_item_to_=22Window=22=2E?= Message-ID: <20150118050957.69912.34420@psf.io> https://hg.python.org/cpython/rev/8c0e5b507794 changeset: 94200:8c0e5b507794 branch: 3.4 parent: 94197:65ac2b992673 user: Ned Deily date: Sat Jan 17 21:06:27 2015 -0800 summary: Issue #23180: Rename IDLE "Windows" menu item to "Window". Patch by Al Sweigart. files: Doc/library/idle.rst | 4 ++-- Lib/idlelib/EditorWindow.py | 5 +---- Lib/idlelib/PyShell.py | 5 +---- Lib/idlelib/help.txt | 2 +- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 6 files changed, 9 insertions(+), 11 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -258,8 +258,8 @@ Open a pane at the top of the edit window which shows the block context of the code which has scrolled above the top of the window. -Windows menu (Shell and Editor) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Window menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Zoom Height Toggles the window between normal size and maximum height. The initial size diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -439,13 +439,10 @@ ("format", "F_ormat"), ("run", "_Run"), ("options", "_Options"), - ("windows", "_Windows"), + ("windows", "_Window"), ("help", "_Help"), ] - if sys.platform == "darwin": - menu_specs[-2] = ("windows", "_Window") - def createmenubar(self): mbar = self.menubar diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -848,13 +848,10 @@ ("edit", "_Edit"), ("debug", "_Debug"), ("options", "_Options"), - ("windows", "_Windows"), + ("windows", "_Window"), ("help", "_Help"), ] - if sys.platform == "darwin": - menu_specs[-2] = ("windows", "_Window") - # New classes from idlelib.IdleHistory import History diff --git a/Lib/idlelib/help.txt b/Lib/idlelib/help.txt --- a/Lib/idlelib/help.txt +++ b/Lib/idlelib/help.txt @@ -138,7 +138,7 @@ window. This is not present in the Shell window only the Editor window. -Windows Menu (Shell and Editor): +Window Menu (Shell and Editor): Zoom Height -- Toggles the window between normal size (40x80 initial setting) and maximum height. The initial size is in the Configure diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1320,6 +1320,7 @@ Kalle Svensson Andrew Svetlov Paul Swartz +Al Sweigart Thenault Sylvain P?ter Szab? John Szakmeister diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -272,6 +272,9 @@ - Issue #21986: Code objects are not normally pickled by the pickle module. To match this, they are no longer pickled when running under Idle. +- Issue #23180: Rename IDLE "Windows" menu item to "Window". + Patch by Al Sweigart. + Tests ----- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 06:10:34 2015 From: python-checkins at python.org (ned.deily) Date: Sun, 18 Jan 2015 05:10:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323180=3A_merge_from_3=2E4?= Message-ID: <20150118050958.103307.68596@psf.io> https://hg.python.org/cpython/rev/af092c1d3747 changeset: 94201:af092c1d3747 parent: 94198:2d71d0f954fb parent: 94200:8c0e5b507794 user: Ned Deily date: Sat Jan 17 21:09:16 2015 -0800 summary: Issue #23180: merge from 3.4 files: Doc/library/idle.rst | 4 ++-- Lib/idlelib/EditorWindow.py | 5 +---- Lib/idlelib/PyShell.py | 5 +---- Lib/idlelib/help.txt | 2 +- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 6 files changed, 9 insertions(+), 11 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -258,8 +258,8 @@ Open a pane at the top of the edit window which shows the block context of the code which has scrolled above the top of the window. -Windows menu (Shell and Editor) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Window menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Zoom Height Toggles the window between normal size and maximum height. The initial size diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -439,13 +439,10 @@ ("format", "F_ormat"), ("run", "_Run"), ("options", "_Options"), - ("windows", "_Windows"), + ("windows", "_Window"), ("help", "_Help"), ] - if sys.platform == "darwin": - menu_specs[-2] = ("windows", "_Window") - def createmenubar(self): mbar = self.menubar diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -848,13 +848,10 @@ ("edit", "_Edit"), ("debug", "_Debug"), ("options", "_Options"), - ("windows", "_Windows"), + ("windows", "_Window"), ("help", "_Help"), ] - if sys.platform == "darwin": - menu_specs[-2] = ("windows", "_Window") - # New classes from idlelib.IdleHistory import History diff --git a/Lib/idlelib/help.txt b/Lib/idlelib/help.txt --- a/Lib/idlelib/help.txt +++ b/Lib/idlelib/help.txt @@ -138,7 +138,7 @@ window. This is not present in the Shell window only the Editor window. -Windows Menu (Shell and Editor): +Window Menu (Shell and Editor): Zoom Height -- Toggles the window between normal size (40x80 initial setting) and maximum height. The initial size is in the Configure diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1343,6 +1343,7 @@ Kalle Svensson Andrew Svetlov Paul Swartz +Al Sweigart Thenault Sylvain P?ter Szab? John Szakmeister diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1374,6 +1374,9 @@ - Issue #17654: Ensure IDLE menus are customized properly on OS X for non-framework builds and for all variants of Tk. +- Issue #23180: Rename IDLE "Windows" menu item to "Window". + Patch by Al Sweigart. + Build ----- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 06:10:35 2015 From: python-checkins at python.org (ned.deily) Date: Sun, 18 Jan 2015 05:10:35 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMTgw?= =?utf-8?q?=3A_Rename_IDLE_=22Windows=22_menu_item_to_=22Window=22=2E?= Message-ID: <20150118050957.93197.51814@psf.io> https://hg.python.org/cpython/rev/9a451aaa8ddb changeset: 94199:9a451aaa8ddb branch: 2.7 parent: 94181:294501835890 user: Ned Deily date: Sat Jan 17 21:03:41 2015 -0800 summary: Issue #23180: Rename IDLE "Windows" menu item to "Window". Patch by Al Sweigart. files: Doc/library/idle.rst | 4 ++-- Lib/idlelib/EditorWindow.py | 5 +---- Lib/idlelib/PyShell.py | 5 +---- Lib/idlelib/help.txt | 2 +- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 6 files changed, 9 insertions(+), 11 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -258,8 +258,8 @@ Open a pane at the top of the edit window which shows the block context of the code which has scrolled above the top of the window. -Windows menu (Shell and Editor) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Window menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Zoom Height Toggles the window between normal size and maximum height. The initial size diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -469,13 +469,10 @@ ("format", "F_ormat"), ("run", "_Run"), ("options", "_Options"), - ("windows", "_Windows"), + ("windows", "_Window"), ("help", "_Help"), ] - if sys.platform == "darwin": - menu_specs[-2] = ("windows", "_Window") - def createmenubar(self): mbar = self.menubar diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -871,13 +871,10 @@ ("edit", "_Edit"), ("debug", "_Debug"), ("options", "_Options"), - ("windows", "_Windows"), + ("windows", "_Window"), ("help", "_Help"), ] - if sys.platform == "darwin": - menu_specs[-2] = ("windows", "_Window") - # New classes from idlelib.IdleHistory import History diff --git a/Lib/idlelib/help.txt b/Lib/idlelib/help.txt --- a/Lib/idlelib/help.txt +++ b/Lib/idlelib/help.txt @@ -100,7 +100,7 @@ which is scrolling off the top or the window. (Not present in Shell window.) -Windows Menu: +Window Menu: Zoom Height -- toggles the window between configured size and maximum height. diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1310,6 +1310,7 @@ Kalle Svensson Andrew Svetlov Paul Swartz +Al Sweigart Thenault Sylvain P?ter Szab? John Szakmeister diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -86,6 +86,9 @@ - Issue #16893: Update Idle doc chapter to match current Idle and add new information. +- Issue #23180: Rename IDLE "Windows" menu item to "Window". + Patch by Al Sweigart. + Build ----- -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Sun Jan 18 09:31:36 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 18 Jan 2015 09:31:36 +0100 Subject: [Python-checkins] Daily reference leaks (2d71d0f954fb): sum=3 Message-ID: results for 2d71d0f954fb on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogfpNKRT', '-x'] From python-checkins at python.org Sun Jan 18 10:18:50 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 09:18:50 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDk4?= =?utf-8?q?=3A_64-bit_dev=5Ft_is_now_supported_in_the_os_module=2E?= Message-ID: <20150118091848.69906.40727@psf.io> https://hg.python.org/cpython/rev/18703ffea2b3 changeset: 94203:18703ffea2b3 branch: 3.4 parent: 94200:8c0e5b507794 user: Serhiy Storchaka date: Sun Jan 18 11:12:11 2015 +0200 summary: Issue #23098: 64-bit dev_t is now supported in the os module. files: Misc/NEWS | 2 + Modules/posixmodule.c | 46 ++++++++++++++++++++++-------- 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #23098: 64-bit dev_t is now supported in the os module. + - Issue #23250: In the http.cookies module, capitalize "HttpOnly" and "Secure" as they are written in the standard. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -623,6 +623,29 @@ #endif /* MS_WINDOWS */ +#if defined(HAVE_MKNOD) && defined(HAVE_MAKEDEV) +static int +_Py_Dev_Converter(PyObject *obj, void *p) +{ +#ifdef HAVE_LONG_LONG + *((dev_t *)p) = PyLong_AsUnsignedLongLong(obj); +#else + *((dev_t *)p) = PyLong_AsUnsignedLong(obj); +#endif + if (PyErr_Occurred()) + return 0; + return 1; +} + +#ifdef HAVE_LONG_LONG +# define _PyLong_FromDev PyLong_FromLongLong +#else +# define _PyLong_FromDev PyLong_FromLong +#endif + +#endif + + #ifdef AT_FDCWD /* * Why the (int) cast? Solaris 10 defines AT_FDCWD as 0xffd19553 (-3041965); @@ -2218,11 +2241,8 @@ #endif #ifdef MS_WINDOWS PyStructSequence_SET_ITEM(v, 2, PyLong_FromUnsignedLong(st->st_dev)); -#elif defined(HAVE_LONG_LONG) - PyStructSequence_SET_ITEM(v, 2, - PyLong_FromLongLong((PY_LONG_LONG)st->st_dev)); -#else - PyStructSequence_SET_ITEM(v, 2, PyLong_FromLong((long)st->st_dev)); +#else + PyStructSequence_SET_ITEM(v, 2, _PyLong_FromDev(st->st_dev)); #endif PyStructSequence_SET_ITEM(v, 3, PyLong_FromLong((long)st->st_nlink)); #if defined(MS_WINDOWS) @@ -8633,16 +8653,16 @@ { path_t path; int mode = 0666; - int device = 0; + dev_t device = 0; int dir_fd = DEFAULT_DIR_FD; int result; PyObject *return_value = NULL; static char *keywords[] = {"path", "mode", "device", "dir_fd", NULL}; memset(&path, 0, sizeof(path)); - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O&|ii$O&:mknod", keywords, + if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O&|iO&$O&:mknod", keywords, path_converter, &path, - &mode, &device, + &mode, _Py_Dev_Converter, &device, #ifdef HAVE_MKNODAT dir_fd_converter, &dir_fd #else @@ -8682,8 +8702,8 @@ static PyObject * posix_major(PyObject *self, PyObject *args) { - int device; - if (!PyArg_ParseTuple(args, "i:major", &device)) + dev_t device; + if (!PyArg_ParseTuple(args, "O&:major", _Py_Dev_Converter, &device)) return NULL; return PyLong_FromLong((long)major(device)); } @@ -8695,8 +8715,8 @@ static PyObject * posix_minor(PyObject *self, PyObject *args) { - int device; - if (!PyArg_ParseTuple(args, "i:minor", &device)) + dev_t device; + if (!PyArg_ParseTuple(args, "O&:minor", _Py_Dev_Converter, &device)) return NULL; return PyLong_FromLong((long)minor(device)); } @@ -8711,7 +8731,7 @@ int major, minor; if (!PyArg_ParseTuple(args, "ii:makedev", &major, &minor)) return NULL; - return PyLong_FromLong((long)makedev(major, minor)); + return _PyLong_FromDev(makedev(major, minor)); } #endif /* device macros */ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 10:18:51 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 09:18:51 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323098=3A_64-bit_dev=5Ft_is_now_supported_in_the?= =?utf-8?q?_os_module=2E?= Message-ID: <20150118091849.103315.1861@psf.io> https://hg.python.org/cpython/rev/fe0fddd6fd21 changeset: 94204:fe0fddd6fd21 parent: 94201:af092c1d3747 parent: 94203:18703ffea2b3 user: Serhiy Storchaka date: Sun Jan 18 11:17:39 2015 +0200 summary: Issue #23098: 64-bit dev_t is now supported in the os module. files: Misc/NEWS | 2 + Modules/posixmodule.c | 109 ++++++++++++++++++----------- 2 files changed, 71 insertions(+), 40 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,8 @@ Library ------- +- Issue #23098: 64-bit dev_t is now supported in the os module. + - Issue #21817: When an exception is raised in a task submitted to a ProcessPoolExecutor, the remote traceback is now displayed in the parent process. Patch by Claudiu Popa. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -617,6 +617,29 @@ #endif /* MS_WINDOWS */ +#if defined(HAVE_MKNOD) && defined(HAVE_MAKEDEV) +static int +_Py_Dev_Converter(PyObject *obj, void *p) +{ +#ifdef HAVE_LONG_LONG + *((dev_t *)p) = PyLong_AsUnsignedLongLong(obj); +#else + *((dev_t *)p) = PyLong_AsUnsignedLong(obj); +#endif + if (PyErr_Occurred()) + return 0; + return 1; +} + +#ifdef HAVE_LONG_LONG +# define _PyLong_FromDev PyLong_FromLongLong +#else +# define _PyLong_FromDev PyLong_FromLong +#endif + +#endif + + #ifdef AT_FDCWD /* * Why the (int) cast? Solaris 10 defines AT_FDCWD as 0xffd19553 (-3041965); @@ -2208,11 +2231,8 @@ #endif #ifdef MS_WINDOWS PyStructSequence_SET_ITEM(v, 2, PyLong_FromUnsignedLong(st->st_dev)); -#elif defined(HAVE_LONG_LONG) - PyStructSequence_SET_ITEM(v, 2, - PyLong_FromLongLong((PY_LONG_LONG)st->st_dev)); -#else - PyStructSequence_SET_ITEM(v, 2, PyLong_FromLong((long)st->st_dev)); +#else + PyStructSequence_SET_ITEM(v, 2, _PyLong_FromDev(st->st_dev)); #endif PyStructSequence_SET_ITEM(v, 3, PyLong_FromLong((long)st->st_nlink)); #if defined(MS_WINDOWS) @@ -2589,6 +2609,15 @@ type = "gid_t" converter = '_Py_Gid_Converter' +class dev_t_converter(CConverter): + type = 'dev_t' + converter = '_Py_Dev_Converter' + +class dev_t_return_converter(unsigned_long_return_converter): + type = 'dev_t' + conversion_fn = '_PyLong_FromDev' + unsigned_cast = '(dev_t)' + class FSConverter_converter(CConverter): type = 'PyObject *' converter = 'PyUnicode_FSConverter' @@ -2639,7 +2668,7 @@ impl_by_reference = True; [python start generated code]*/ -/*[python end generated code: output=da39a3ee5e6b4b0d input=147ba8f52a05aca4]*/ +/*[python end generated code: output=da39a3ee5e6b4b0d input=affe68316f160401]*/ /*[clinic input] @@ -12348,7 +12377,7 @@ path: path_t mode: int=0o600 - device: int=0 + device: dev_t=0 * dir_fd: dir_fd(requires='mknodat')=None @@ -12389,7 +12418,7 @@ {"mknod", (PyCFunction)os_mknod, METH_VARARGS|METH_KEYWORDS, os_mknod__doc__}, static PyObject * -os_mknod_impl(PyModuleDef *module, path_t *path, int mode, int device, int dir_fd); +os_mknod_impl(PyModuleDef *module, path_t *path, int mode, dev_t device, int dir_fd); static PyObject * os_mknod(PyModuleDef *module, PyObject *args, PyObject *kwargs) @@ -12398,12 +12427,12 @@ static char *_keywords[] = {"path", "mode", "device", "dir_fd", NULL}; path_t path = PATH_T_INITIALIZE("mknod", "path", 0, 0); int mode = 384; - int device = 0; + dev_t device = 0; int dir_fd = DEFAULT_DIR_FD; if (!PyArg_ParseTupleAndKeywords(args, kwargs, - "O&|ii$O&:mknod", _keywords, - path_converter, &path, &mode, &device, MKNODAT_DIR_FD_CONVERTER, &dir_fd)) + "O&|iO&$O&:mknod", _keywords, + path_converter, &path, &mode, _Py_Dev_Converter, &device, MKNODAT_DIR_FD_CONVERTER, &dir_fd)) goto exit; return_value = os_mknod_impl(module, &path, mode, device, dir_fd); @@ -12415,8 +12444,8 @@ } static PyObject * -os_mknod_impl(PyModuleDef *module, path_t *path, int mode, int device, int dir_fd) -/*[clinic end generated code: output=c688739c15ca7bbb input=30e02126aba9732e]*/ +os_mknod_impl(PyModuleDef *module, path_t *path, int mode, dev_t device, int dir_fd) +/*[clinic end generated code: output=f71d54eaf9bb6f1a input=ee44531551a4d83b]*/ { int result; @@ -12441,7 +12470,7 @@ /*[clinic input] os.major -> unsigned_int - device: int + device: dev_t / Extracts a device major number from a raw device number. @@ -12457,18 +12486,18 @@ {"major", (PyCFunction)os_major, METH_VARARGS, os_major__doc__}, static unsigned int -os_major_impl(PyModuleDef *module, int device); +os_major_impl(PyModuleDef *module, dev_t device); static PyObject * os_major(PyModuleDef *module, PyObject *args) { PyObject *return_value = NULL; - int device; + dev_t device; unsigned int _return_value; if (!PyArg_ParseTuple(args, - "i:major", - &device)) + "O&:major", + _Py_Dev_Converter, &device)) goto exit; _return_value = os_major_impl(module, device); if ((_return_value == (unsigned int)-1) && PyErr_Occurred()) @@ -12480,8 +12509,8 @@ } static unsigned int -os_major_impl(PyModuleDef *module, int device) -/*[clinic end generated code: output=52e6743300dcf4ad input=ea48820b7e10d310]*/ +os_major_impl(PyModuleDef *module, dev_t device) +/*[clinic end generated code: output=a2d06e908ebf95b5 input=1e16a4d30c4d4462]*/ { return major(device); } @@ -12490,7 +12519,7 @@ /*[clinic input] os.minor -> unsigned_int - device: int + device: dev_t / Extracts a device minor number from a raw device number. @@ -12506,18 +12535,18 @@ {"minor", (PyCFunction)os_minor, METH_VARARGS, os_minor__doc__}, static unsigned int -os_minor_impl(PyModuleDef *module, int device); +os_minor_impl(PyModuleDef *module, dev_t device); static PyObject * os_minor(PyModuleDef *module, PyObject *args) { PyObject *return_value = NULL; - int device; + dev_t device; unsigned int _return_value; if (!PyArg_ParseTuple(args, - "i:minor", - &device)) + "O&:minor", + _Py_Dev_Converter, &device)) goto exit; _return_value = os_minor_impl(module, device); if ((_return_value == (unsigned int)-1) && PyErr_Occurred()) @@ -12529,15 +12558,15 @@ } static unsigned int -os_minor_impl(PyModuleDef *module, int device) -/*[clinic end generated code: output=aebe4bd7f455b755 input=089733ebbf9754e8]*/ +os_minor_impl(PyModuleDef *module, dev_t device) +/*[clinic end generated code: output=6332287ee3f006e2 input=0842c6d23f24c65e]*/ { return minor(device); } /*[clinic input] -os.makedev -> unsigned_int +os.makedev -> dev_t major: int minor: int @@ -12555,7 +12584,7 @@ #define OS_MAKEDEV_METHODDEF \ {"makedev", (PyCFunction)os_makedev, METH_VARARGS, os_makedev__doc__}, -static unsigned int +static dev_t os_makedev_impl(PyModuleDef *module, int major, int minor); static PyObject * @@ -12564,24 +12593,24 @@ PyObject *return_value = NULL; int major; int minor; - unsigned int _return_value; + dev_t _return_value; if (!PyArg_ParseTuple(args, "ii:makedev", &major, &minor)) goto exit; _return_value = os_makedev_impl(module, major, minor); - if ((_return_value == (unsigned int)-1) && PyErr_Occurred()) - goto exit; - return_value = PyLong_FromUnsignedLong((unsigned long)_return_value); - -exit: - return return_value; -} - -static unsigned int + if ((_return_value == (dev_t)-1) && PyErr_Occurred()) + goto exit; + return_value = _PyLong_FromDev(_return_value); + +exit: + return return_value; +} + +static dev_t os_makedev_impl(PyModuleDef *module, int major, int minor) -/*[clinic end generated code: output=5cb79d9c9eac58b0 input=f55bf7cffb028a08]*/ +/*[clinic end generated code: output=38e9a9774c96511a input=4b9fd8fc73cbe48f]*/ { return makedev(major, minor); } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 10:18:51 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 09:18:51 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMDk4?= =?utf-8?q?=3A_64-bit_dev=5Ft_is_now_supported_in_the_os_module=2E?= Message-ID: <20150118091848.93179.34981@psf.io> https://hg.python.org/cpython/rev/7ee09e2fec13 changeset: 94202:7ee09e2fec13 branch: 2.7 parent: 94199:9a451aaa8ddb user: Serhiy Storchaka date: Sun Jan 18 11:11:25 2015 +0200 summary: Issue #23098: 64-bit dev_t is now supported in the os module. files: Misc/NEWS | 2 + Modules/posixmodule.c | 54 ++++++++++++++++++++++++------ 2 files changed, 44 insertions(+), 12 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Library ------- +- Issue #23098: 64-bit dev_t is now supported in the os module. + - Issue #23063: In the disutils' check command, fix parsing of reST with code or code-block directives. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -474,6 +474,36 @@ #endif /* MS_WINDOWS */ +#if defined(HAVE_MKNOD) && defined(HAVE_MAKEDEV) +static int +_Py_Dev_Converter(PyObject *obj, void *p) +{ +#ifdef HAVE_LONG_LONG + *((dev_t *)p) = PyLong_AsUnsignedLongLong(obj); +#else + *((dev_t *)p) = PyLong_AsUnsignedLong(obj); +#endif + if (PyErr_Occurred()) + return 0; + return 1; +} + +#ifdef HAVE_LONG_LONG +static PyObject * +_PyInt_FromDev(PY_LONG_LONG v) +{ + if (LONG_MIN <= v && v <= LONG_MAX) + return PyInt_FromLong((long)v); + else + return PyLong_FromLongLong(v); +} +#else +# define _PyInt_FromDev PyInt_FromLong +#endif + +#endif + + #if defined _MSC_VER && _MSC_VER >= 1400 /* Microsoft CRT in VS2005 and higher will verify that a filehandle is * valid and raise an assertion if it isn't. @@ -1426,11 +1456,10 @@ #else PyStructSequence_SET_ITEM(v, 1, PyInt_FromLong((long)st->st_ino)); #endif -#if defined(HAVE_LONG_LONG) && !defined(MS_WINDOWS) - PyStructSequence_SET_ITEM(v, 2, - PyLong_FromLongLong((PY_LONG_LONG)st->st_dev)); +#ifdef MS_WINDOWS + PyStructSequence_SET_ITEM(v, 2, PyLong_FromUnsignedLong(st->st_dev)); #else - PyStructSequence_SET_ITEM(v, 2, PyInt_FromLong((long)st->st_dev)); + PyStructSequence_SET_ITEM(v, 2, _PyInt_FromDev(st->st_dev)); #endif PyStructSequence_SET_ITEM(v, 3, PyInt_FromLong((long)st->st_nlink)); #if defined(MS_WINDOWS) @@ -7009,9 +7038,11 @@ { char *filename; int mode = 0600; - int device = 0; + dev_t device = 0; int res; - if (!PyArg_ParseTuple(args, "s|ii:mknod", &filename, &mode, &device)) + if (!PyArg_ParseTuple(args, "s|iO&:mknod", + &filename, &mode, + _Py_Dev_Converter, &device)) return NULL; Py_BEGIN_ALLOW_THREADS res = mknod(filename, mode, device); @@ -7031,8 +7062,8 @@ static PyObject * posix_major(PyObject *self, PyObject *args) { - int device; - if (!PyArg_ParseTuple(args, "i:major", &device)) + dev_t device; + if (!PyArg_ParseTuple(args, "O&:major", _Py_Dev_Converter, &device)) return NULL; return PyInt_FromLong((long)major(device)); } @@ -7044,8 +7075,8 @@ static PyObject * posix_minor(PyObject *self, PyObject *args) { - int device; - if (!PyArg_ParseTuple(args, "i:minor", &device)) + dev_t device; + if (!PyArg_ParseTuple(args, "O&:minor", _Py_Dev_Converter, &device)) return NULL; return PyInt_FromLong((long)minor(device)); } @@ -7060,7 +7091,7 @@ int major, minor; if (!PyArg_ParseTuple(args, "ii:makedev", &major, &minor)) return NULL; - return PyInt_FromLong((long)makedev(major, minor)); + return _PyInt_FromDev(makedev(major, minor)); } #endif /* device macros */ @@ -8522,7 +8553,6 @@ { PyObject *d = NULL; size_t i; - qsort(table, tablesize, sizeof(struct constdef), cmp_constdefs); d = PyDict_New(); if (d == NULL) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 10:45:18 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 09:45:18 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMTgx?= =?utf-8?b?OiBNb3JlICJjb2RlcG9pbnQiIC0+ICJjb2RlIHBvaW50Ii4=?= Message-ID: <20150118094515.118090.28610@psf.io> https://hg.python.org/cpython/rev/2db41d551a4f changeset: 94207:2db41d551a4f branch: 2.7 parent: 94202:7ee09e2fec13 user: Serhiy Storchaka date: Sun Jan 18 11:42:50 2015 +0200 summary: Issue #23181: More "codepoint" -> "code point". files: Doc/library/sgmllib.rst | 2 +- Lib/htmlentitydefs.py | 4 ++-- Lib/test/test_multibytecodec.py | 2 +- Lib/test/test_multibytecodec_support.py | 2 +- Lib/test/test_stringprep.py | 2 +- Lib/test/test_unicode.py | 4 ++-- Modules/cjkcodecs/_codecs_cn.c | 2 +- Modules/cjkcodecs/_codecs_hk.c | 2 +- Modules/cjkcodecs/_codecs_kr.c | 2 +- Modules/cjkcodecs/cjkcodecs.h | 4 ++-- Objects/unicodeobject.c | 8 ++++---- 11 files changed, 17 insertions(+), 17 deletions(-) diff --git a/Doc/library/sgmllib.rst b/Doc/library/sgmllib.rst --- a/Doc/library/sgmllib.rst +++ b/Doc/library/sgmllib.rst @@ -153,7 +153,7 @@ .. method:: SGMLParser.convert_codepoint(codepoint) - Convert a codepoint to a :class:`str` value. Encodings can be handled here if + Convert a code point to a :class:`str` value. Encodings can be handled here if appropriate, though the rest of :mod:`sgmllib` is oblivious on this matter. .. versionadded:: 2.5 diff --git a/Lib/htmlentitydefs.py b/Lib/htmlentitydefs.py --- a/Lib/htmlentitydefs.py +++ b/Lib/htmlentitydefs.py @@ -1,6 +1,6 @@ """HTML character entity references.""" -# maps the HTML entity name to the Unicode codepoint +# maps the HTML entity name to the Unicode code point name2codepoint = { 'AElig': 0x00c6, # latin capital letter AE = latin capital ligature AE, U+00C6 ISOlat1 'Aacute': 0x00c1, # latin capital letter A with acute, U+00C1 ISOlat1 @@ -256,7 +256,7 @@ 'zwnj': 0x200c, # zero width non-joiner, U+200C NEW RFC 2070 } -# maps the Unicode codepoint to the HTML entity name +# maps the Unicode code point to the HTML entity name codepoint2name = {} # maps the HTML entity name to the character diff --git a/Lib/test/test_multibytecodec.py b/Lib/test/test_multibytecodec.py --- a/Lib/test/test_multibytecodec.py +++ b/Lib/test/test_multibytecodec.py @@ -72,7 +72,7 @@ self.assertEqual(encoder.reset(), None) def test_stateful(self): - # jisx0213 encoder is stateful for a few codepoints. eg) + # jisx0213 encoder is stateful for a few code points. eg) # U+00E6 => A9DC # U+00E6 U+0300 => ABC4 # U+0300 => ABDC diff --git a/Lib/test/test_multibytecodec_support.py b/Lib/test/test_multibytecodec_support.py --- a/Lib/test/test_multibytecodec_support.py +++ b/Lib/test/test_multibytecodec_support.py @@ -20,7 +20,7 @@ roundtriptest = 1 # set if roundtrip is possible with unicode has_iso10646 = 0 # set if this encoding contains whole iso10646 map xmlcharnametest = None # string to test xmlcharrefreplace - unmappedunicode = u'\udeee' # a unicode codepoint that is not mapped. + unmappedunicode = u'\udeee' # a unicode code point that is not mapped. def setUp(self): if self.codec is None: diff --git a/Lib/test/test_stringprep.py b/Lib/test/test_stringprep.py --- a/Lib/test/test_stringprep.py +++ b/Lib/test/test_stringprep.py @@ -1,5 +1,5 @@ # To fully test this module, we would need a copy of the stringprep tables. -# Since we don't have them, this test checks only a few codepoints. +# Since we don't have them, this test checks only a few code points. import unittest from test import test_support diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -874,9 +874,9 @@ def test_utf8_decode_invalid_sequences(self): # continuation bytes in a sequence of 2, 3, or 4 bytes continuation_bytes = map(chr, range(0x80, 0xC0)) - # start bytes of a 2-byte sequence equivalent to codepoints < 0x7F + # start bytes of a 2-byte sequence equivalent to code points < 0x7F invalid_2B_seq_start_bytes = map(chr, range(0xC0, 0xC2)) - # start bytes of a 4-byte sequence equivalent to codepoints > 0x10FFFF + # start bytes of a 4-byte sequence equivalent to code points > 0x10FFFF invalid_4B_seq_start_bytes = map(chr, range(0xF5, 0xF8)) invalid_start_bytes = ( continuation_bytes + invalid_2B_seq_start_bytes + diff --git a/Modules/cjkcodecs/_codecs_cn.c b/Modules/cjkcodecs/_codecs_cn.c --- a/Modules/cjkcodecs/_codecs_cn.c +++ b/Modules/cjkcodecs/_codecs_cn.c @@ -15,7 +15,7 @@ #undef hz #endif -/* GBK and GB2312 map differently in few codepoints that are listed below: +/* GBK and GB2312 map differently in few code points that are listed below: * * gb2312 gbk * A1A4 U+30FB KATAKANA MIDDLE DOT U+00B7 MIDDLE DOT diff --git a/Modules/cjkcodecs/_codecs_hk.c b/Modules/cjkcodecs/_codecs_hk.c --- a/Modules/cjkcodecs/_codecs_hk.c +++ b/Modules/cjkcodecs/_codecs_hk.c @@ -164,7 +164,7 @@ default: return 2; } - NEXT(2, 2) /* all decoded codepoints are pairs, above. */ + NEXT(2, 2) /* all decoded code points are pairs, above. */ } return 0; diff --git a/Modules/cjkcodecs/_codecs_kr.c b/Modules/cjkcodecs/_codecs_kr.c --- a/Modules/cjkcodecs/_codecs_kr.c +++ b/Modules/cjkcodecs/_codecs_kr.c @@ -64,7 +64,7 @@ OUT1(EUCKR_JAMO_FIRSTBYTE) OUT2(EUCKR_JAMO_FILLER) - /* All codepoints in CP949 extension are in unicode + /* All code points in CP949 extension are in unicode * Hangul Syllable area. */ assert(0xac00 <= c && c <= 0xd7a3); c -= 0xac00; diff --git a/Modules/cjkcodecs/cjkcodecs.h b/Modules/cjkcodecs/cjkcodecs.h --- a/Modules/cjkcodecs/cjkcodecs.h +++ b/Modules/cjkcodecs/cjkcodecs.h @@ -12,10 +12,10 @@ #include "multibytecodec.h" -/* a unicode "undefined" codepoint */ +/* a unicode "undefined" code point */ #define UNIINV 0xFFFE -/* internal-use DBCS codepoints which aren't used by any charsets */ +/* internal-use DBCS code points which aren't used by any charsets */ #define NOCHAR 0xFFFF #define MULTIC 0xFFFE #define DBCINV 0xFFFD diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -2039,7 +2039,7 @@ see http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf (table 3-7) and http://www.rfc-editor.org/rfc/rfc3629.txt Uncomment the 2 lines below to make them invalid, - codepoints: d800-dfff; UTF-8: \xed\xa0\x80-\xed\xbf\xbf. */ + code points: d800-dfff; UTF-8: \xed\xa0\x80-\xed\xbf\xbf. */ if ((s[1] & 0xc0) != 0x80 || (s[2] & 0xc0) != 0x80 || ((unsigned char)s[0] == 0xE0 && @@ -2337,7 +2337,7 @@ } /* On narrow builds we split characters outside the BMP into two - codepoints => count how much extra space we need. */ + code points => count how much extra space we need. */ #ifndef Py_UNICODE_WIDE for (qq = q; e - qq >= 4; qq += 4) if (qq[iorder[2]] != 0 || qq[iorder[3]] != 0) @@ -2372,7 +2372,7 @@ if (ch >= 0x110000) { - errmsg = "codepoint not in range(0x110000)"; + errmsg = "code point not in range(0x110000)"; startinpos = ((const char *)q)-starts; endinpos = startinpos+4; goto utf32Error; @@ -2449,7 +2449,7 @@ p += 4; \ } while(0) - /* In narrow builds we can output surrogate pairs as one codepoint, + /* In narrow builds we can output surrogate pairs as one code point, so we need less space. */ #ifndef Py_UNICODE_WIDE for (i = pairs = 0; i < size-1; i++) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 10:45:18 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 09:45:18 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323181=3A_More_=22codepoint=22_-=3E_=22code_poin?= =?utf-8?b?dCIu?= Message-ID: <20150118094515.93179.38706@psf.io> https://hg.python.org/cpython/rev/c79abee84a39 changeset: 94206:c79abee84a39 parent: 94204:fe0fddd6fd21 parent: 94205:0353c7e5e0c2 user: Serhiy Storchaka date: Sun Jan 18 11:33:31 2015 +0200 summary: Issue #23181: More "codepoint" -> "code point". files: Doc/library/json.rst | 2 +- Lib/codecs.py | 2 +- Lib/email/message.py | 2 +- Lib/html/entities.py | 4 ++-- Lib/test/multibytecodec_support.py | 2 +- Lib/test/test_html.py | 4 ++-- Lib/test/test_multibytecodec.py | 2 +- Lib/test/test_stringprep.py | 2 +- Lib/test/test_unicode.py | 4 ++-- Modules/cjkcodecs/_codecs_cn.c | 2 +- Modules/cjkcodecs/_codecs_hk.c | 2 +- Modules/cjkcodecs/_codecs_kr.c | 2 +- Modules/cjkcodecs/cjkcodecs.h | 4 ++-- Modules/unicodedata.c | 14 +++++++------- Objects/unicodeobject.c | 4 ++-- Python/sysmodule.c | 2 +- Tools/unicode/gencodec.py | 2 +- 17 files changed, 28 insertions(+), 28 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -514,7 +514,7 @@ that don't correspond to valid Unicode characters (e.g. unpaired UTF-16 surrogates), but it does note that they may cause interoperability problems. By default, this module accepts and outputs (when present in the original -:class:`str`) codepoints for such sequences. +:class:`str`) code points for such sequences. Infinite and NaN Number Values diff --git a/Lib/codecs.py b/Lib/codecs.py --- a/Lib/codecs.py +++ b/Lib/codecs.py @@ -124,7 +124,7 @@ Python will use the official U+FFFD REPLACEMENT CHARACTER for the builtin Unicode codecs on decoding and '?' on encoding. - 'surrogateescape' - replace with private codepoints U+DCnn. + 'surrogateescape' - replace with private code points U+DCnn. 'xmlcharrefreplace' - Replace with the appropriate XML character reference (only for encoding). 'backslashreplace' - Replace with backslashed escape sequences diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -273,7 +273,7 @@ bpayload = payload.encode('ascii') except UnicodeError: # This won't happen for RFC compliant messages (messages - # containing only ASCII codepoints in the unicode input). + # containing only ASCII code points in the unicode input). # If it does happen, turn the string into bytes in a way # guaranteed not to fail. bpayload = payload.encode('raw-unicode-escape') diff --git a/Lib/html/entities.py b/Lib/html/entities.py --- a/Lib/html/entities.py +++ b/Lib/html/entities.py @@ -3,7 +3,7 @@ __all__ = ['html5', 'name2codepoint', 'codepoint2name', 'entitydefs'] -# maps the HTML entity name to the Unicode codepoint +# maps the HTML entity name to the Unicode code point name2codepoint = { 'AElig': 0x00c6, # latin capital letter AE = latin capital ligature AE, U+00C6 ISOlat1 'Aacute': 0x00c1, # latin capital letter A with acute, U+00C1 ISOlat1 @@ -2495,7 +2495,7 @@ 'zwnj;': '\u200c', } -# maps the Unicode codepoint to the HTML entity name +# maps the Unicode code point to the HTML entity name codepoint2name = {} # maps the HTML entity name to the character diff --git a/Lib/test/multibytecodec_support.py b/Lib/test/multibytecodec_support.py --- a/Lib/test/multibytecodec_support.py +++ b/Lib/test/multibytecodec_support.py @@ -21,7 +21,7 @@ roundtriptest = 1 # set if roundtrip is possible with unicode has_iso10646 = 0 # set if this encoding contains whole iso10646 map xmlcharnametest = None # string to test xmlcharrefreplace - unmappedunicode = '\udeee' # a unicode codepoint that is not mapped. + unmappedunicode = '\udeee' # a unicode code point that is not mapped. def setUp(self): if self.codec is None: diff --git a/Lib/test/test_html.py b/Lib/test/test_html.py --- a/Lib/test/test_html.py +++ b/Lib/test/test_html.py @@ -48,10 +48,10 @@ check(s % num, char) for end in [' ', 'X']: check((s+end) % num, char+end) - # check invalid codepoints + # check invalid code points for cp in [0xD800, 0xDB00, 0xDC00, 0xDFFF, 0x110000]: check_num(cp, '\uFFFD') - # check more invalid codepoints + # check more invalid code points for cp in [0x1, 0xb, 0xe, 0x7f, 0xfffe, 0xffff, 0x10fffe, 0x10ffff]: check_num(cp, '') # check invalid numbers diff --git a/Lib/test/test_multibytecodec.py b/Lib/test/test_multibytecodec.py --- a/Lib/test/test_multibytecodec.py +++ b/Lib/test/test_multibytecodec.py @@ -80,7 +80,7 @@ self.assertEqual(encoder.reset(), None) def test_stateful(self): - # jisx0213 encoder is stateful for a few codepoints. eg) + # jisx0213 encoder is stateful for a few code points. eg) # U+00E6 => A9DC # U+00E6 U+0300 => ABC4 # U+0300 => ABDC diff --git a/Lib/test/test_stringprep.py b/Lib/test/test_stringprep.py --- a/Lib/test/test_stringprep.py +++ b/Lib/test/test_stringprep.py @@ -1,5 +1,5 @@ # To fully test this module, we would need a copy of the stringprep tables. -# Since we don't have them, this test checks only a few codepoints. +# Since we don't have them, this test checks only a few code points. import unittest from test import support diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1470,9 +1470,9 @@ def test_utf8_decode_invalid_sequences(self): # continuation bytes in a sequence of 2, 3, or 4 bytes continuation_bytes = [bytes([x]) for x in range(0x80, 0xC0)] - # start bytes of a 2-byte sequence equivalent to codepoints < 0x7F + # start bytes of a 2-byte sequence equivalent to code points < 0x7F invalid_2B_seq_start_bytes = [bytes([x]) for x in range(0xC0, 0xC2)] - # start bytes of a 4-byte sequence equivalent to codepoints > 0x10FFFF + # start bytes of a 4-byte sequence equivalent to code points > 0x10FFFF invalid_4B_seq_start_bytes = [bytes([x]) for x in range(0xF5, 0xF8)] invalid_start_bytes = ( continuation_bytes + invalid_2B_seq_start_bytes + diff --git a/Modules/cjkcodecs/_codecs_cn.c b/Modules/cjkcodecs/_codecs_cn.c --- a/Modules/cjkcodecs/_codecs_cn.c +++ b/Modules/cjkcodecs/_codecs_cn.c @@ -15,7 +15,7 @@ #undef hz #endif -/* GBK and GB2312 map differently in few codepoints that are listed below: +/* GBK and GB2312 map differently in few code points that are listed below: * * gb2312 gbk * A1A4 U+30FB KATAKANA MIDDLE DOT U+00B7 MIDDLE DOT diff --git a/Modules/cjkcodecs/_codecs_hk.c b/Modules/cjkcodecs/_codecs_hk.c --- a/Modules/cjkcodecs/_codecs_hk.c +++ b/Modules/cjkcodecs/_codecs_hk.c @@ -171,7 +171,7 @@ default: return 1; } - NEXT_IN(2); /* all decoded codepoints are pairs, above. */ + NEXT_IN(2); /* all decoded code points are pairs, above. */ } return 0; diff --git a/Modules/cjkcodecs/_codecs_kr.c b/Modules/cjkcodecs/_codecs_kr.c --- a/Modules/cjkcodecs/_codecs_kr.c +++ b/Modules/cjkcodecs/_codecs_kr.c @@ -69,7 +69,7 @@ OUTBYTE1(EUCKR_JAMO_FIRSTBYTE); OUTBYTE2(EUCKR_JAMO_FILLER); - /* All codepoints in CP949 extension are in unicode + /* All code points in CP949 extension are in unicode * Hangul Syllable area. */ assert(0xac00 <= c && c <= 0xd7a3); c -= 0xac00; diff --git a/Modules/cjkcodecs/cjkcodecs.h b/Modules/cjkcodecs/cjkcodecs.h --- a/Modules/cjkcodecs/cjkcodecs.h +++ b/Modules/cjkcodecs/cjkcodecs.h @@ -12,10 +12,10 @@ #include "multibytecodec.h" -/* a unicode "undefined" codepoint */ +/* a unicode "undefined" code point */ #define UNIINV 0xFFFE -/* internal-use DBCS codepoints which aren't used by any charsets */ +/* internal-use DBCS code points which aren't used by any charsets */ #define NOCHAR 0xFFFF #define MULTIC 0xFFFE #define DBCINV 0xFFFD diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c --- a/Modules/unicodedata.c +++ b/Modules/unicodedata.c @@ -976,7 +976,7 @@ (0x2B740 <= code && code <= 0x2B81D); /* CJK Ideograph Extension D */ } -/* macros used to determine if the given codepoint is in the PUA range that +/* macros used to determine if the given code point is in the PUA range that * we are using to store aliases and named sequences */ #define IS_ALIAS(cp) ((cp >= aliases_start) && (cp < aliases_end)) #define IS_NAMED_SEQ(cp) ((cp >= named_sequences_start) && \ @@ -986,7 +986,7 @@ _getucname(PyObject *self, Py_UCS4 code, char* buffer, int buflen, int with_alias_and_seq) { - /* Find the name associated with the given codepoint. + /* Find the name associated with the given code point. * If with_alias_and_seq is 1, check for names in the Private Use Area 15 * that we are using for aliases and named sequences. */ int offset; @@ -997,7 +997,7 @@ if (code >= 0x110000) return 0; - /* XXX should we just skip all the codepoints in the PUAs here? */ + /* XXX should we just skip all the code points in the PUAs here? */ if (!with_alias_and_seq && (IS_ALIAS(code) || IS_NAMED_SEQ(code))) return 0; @@ -1125,8 +1125,8 @@ /* check if named sequences are allowed */ if (!with_named_seq && IS_NAMED_SEQ(cp)) return 0; - /* if the codepoint is in the PUA range that we use for aliases, - * convert it to obtain the right codepoint */ + /* if the code point is in the PUA range that we use for aliases, + * convert it to obtain the right code point */ if (IS_ALIAS(cp)) *code = name_aliases[cp-aliases_start]; else @@ -1138,9 +1138,9 @@ _getcode(PyObject* self, const char* name, int namelen, Py_UCS4* code, int with_named_seq) { - /* Return the codepoint associated with the given name. + /* Return the code point associated with the given name. * Named aliases are resolved too (unless self != NULL (i.e. we are using - * 3.2.0)). If with_named_seq is 1, returns the PUA codepoint that we are + * 3.2.0)). If with_named_seq is 1, returns the PUA code point that we are * using for the named sequence, and the caller must then convert it. */ unsigned int h, v; unsigned int mask = code_size-1; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5013,7 +5013,7 @@ } if (Py_UNICODE_IS_SURROGATE(ch)) { - errmsg = "codepoint in surrogate code point range(0xd800, 0xe000)"; + errmsg = "code point in surrogate code point range(0xd800, 0xe000)"; startinpos = ((const char *)q) - starts; endinpos = startinpos + 4; } @@ -5032,7 +5032,7 @@ q += 4; continue; } - errmsg = "codepoint not in range(0x110000)"; + errmsg = "code point not in range(0x110000)"; startinpos = ((const char *)q) - starts; endinpos = startinpos + 4; } diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1379,7 +1379,7 @@ implementation -- Python implementation information.\n\ int_info -- a struct sequence with information about the int implementation.\n\ maxsize -- the largest supported length of containers.\n\ -maxunicode -- the value of the largest Unicode codepoint\n\ +maxunicode -- the value of the largest Unicode code point\n\ platform -- platform identifier\n\ prefix -- prefix used to find the Python library\n\ thread_info -- a struct sequence with information about the thread implementation.\n\ diff --git a/Tools/unicode/gencodec.py b/Tools/unicode/gencodec.py --- a/Tools/unicode/gencodec.py +++ b/Tools/unicode/gencodec.py @@ -34,7 +34,7 @@ # Standard undefined Unicode code point UNI_UNDEFINED = chr(0xFFFE) -# Placeholder for a missing codepoint +# Placeholder for a missing code point MISSING_CODE = -1 mapRE = re.compile('((?:0x[0-9a-fA-F]+\+?)+)' -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 10:45:18 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 09:45:18 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMTgx?= =?utf-8?b?OiBNb3JlICJjb2RlcG9pbnQiIC0+ICJjb2RlIHBvaW50Ii4=?= Message-ID: <20150118094514.118094.85890@psf.io> https://hg.python.org/cpython/rev/0353c7e5e0c2 changeset: 94205:0353c7e5e0c2 branch: 3.4 parent: 94203:18703ffea2b3 user: Serhiy Storchaka date: Sun Jan 18 11:28:37 2015 +0200 summary: Issue #23181: More "codepoint" -> "code point". files: Doc/c-api/unicode.rst | 2 +- Doc/library/codecs.rst | 12 ++++++------ Doc/library/email.mime.rst | 2 +- Doc/library/functions.rst | 2 +- Doc/library/html.entities.rst | 4 ++-- Doc/library/json.rst | 2 +- Doc/tutorial/datastructures.rst | 2 +- Doc/whatsnew/3.3.rst | 18 +++++++++--------- Lib/codecs.py | 2 +- Lib/email/message.py | 2 +- Lib/html/entities.py | 4 ++-- Lib/test/multibytecodec_support.py | 2 +- Lib/test/test_html.py | 4 ++-- Lib/test/test_multibytecodec.py | 2 +- Lib/test/test_stringprep.py | 2 +- Lib/test/test_unicode.py | 4 ++-- Modules/cjkcodecs/_codecs_cn.c | 2 +- Modules/cjkcodecs/_codecs_hk.c | 2 +- Modules/cjkcodecs/_codecs_kr.c | 2 +- Modules/cjkcodecs/cjkcodecs.h | 4 ++-- Modules/unicodedata.c | 14 +++++++------- Objects/unicodeobject.c | 4 ++-- Python/sysmodule.c | 2 +- Tools/unicode/gencodec.py | 2 +- 24 files changed, 49 insertions(+), 49 deletions(-) diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -1134,7 +1134,7 @@ mark (U+FEFF). In the other two modes, no BOM mark is prepended. If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output - as a single codepoint. + as a single code point. Return *NULL* if an exception was raised by the codec. diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -827,7 +827,7 @@ Encodings and Unicode --------------------- -Strings are stored internally as sequences of codepoints in +Strings are stored internally as sequences of code points in range ``0x0``-``0x10FFFF``. (See :pep:`393` for more details about the implementation.) Once a string object is used outside of CPU and memory, endianness @@ -838,23 +838,23 @@ collectivity referred to as :term:`text encodings `. The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps -the codepoints 0-255 to the bytes ``0x0``-``0xff``, which means that a string -object that contains codepoints above ``U+00FF`` can't be encoded with this +the code points 0-255 to the bytes ``0x0``-``0xff``, which means that a string +object that contains code points above ``U+00FF`` can't be encoded with this codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks like the following (although the details of the error message may differ): ``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in position 3: ordinal not in range(256)``. There's another group of encodings (the so called charmap encodings) that choose -a different subset of all Unicode code points and how these codepoints are +a different subset of all Unicode code points and how these code points are mapped to the bytes ``0x0``-``0xff``. To see how this is done simply open e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on Windows). There's a string constant with 256 characters that shows you which character is mapped to which byte value. -All of these encodings can only encode 256 of the 1114112 codepoints +All of these encodings can only encode 256 of the 1114112 code points defined in Unicode. A simple and straightforward way that can store each Unicode -code point, is to store each codepoint as four consecutive bytes. There are two +code point, is to store each code point as four consecutive bytes. There are two possibilities: store the bytes in big endian or in little endian order. These two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their disadvantage is that if e.g. you use ``UTF-32-BE`` on a little endian machine you diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -194,7 +194,7 @@ minor type and defaults to :mimetype:`plain`. *_charset* is the character set of the text and is passed as an argument to the :class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults - to ``us-ascii`` if the string contains only ``ascii`` codepoints, and + to ``us-ascii`` if the string contains only ``ascii`` code points, and ``utf-8`` otherwise. Unless the *_charset* argument is explicitly set to ``None``, the diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -156,7 +156,7 @@ .. function:: chr(i) - Return the string representing a character whose Unicode codepoint is the integer + Return the string representing a character whose Unicode code point is the integer *i*. For example, ``chr(97)`` returns the string ``'a'``. This is the inverse of :func:`ord`. The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in base 16). :exc:`ValueError` will be raised if *i* is diff --git a/Doc/library/html.entities.rst b/Doc/library/html.entities.rst --- a/Doc/library/html.entities.rst +++ b/Doc/library/html.entities.rst @@ -33,12 +33,12 @@ .. data:: name2codepoint - A dictionary that maps HTML entity names to the Unicode codepoints. + A dictionary that maps HTML entity names to the Unicode code points. .. data:: codepoint2name - A dictionary that maps Unicode codepoints to HTML entity names. + A dictionary that maps Unicode code points to HTML entity names. .. rubric:: Footnotes diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -512,7 +512,7 @@ that don't correspond to valid Unicode characters (e.g. unpaired UTF-16 surrogates), but it does note that they may cause interoperability problems. By default, this module accepts and outputs (when present in the original -:class:`str`) codepoints for such sequences. +:class:`str`) code points for such sequences. Infinite and NaN Number Values diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -684,7 +684,7 @@ all items of two sequences compare equal, the sequences are considered equal. If one sequence is an initial sub-sequence of the other, the shorter sequence is the smaller (lesser) one. Lexicographical ordering for strings uses the Unicode -codepoint number to order individual characters. Some examples of comparisons +code point number to order individual characters. Some examples of comparisons between sequences of the same type:: (1, 2, 3) < (1, 2, 4) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -228,7 +228,7 @@ Changes introduced by :pep:`393` are the following: -* Python now always supports the full range of Unicode codepoints, including +* Python now always supports the full range of Unicode code points, including non-BMP ones (i.e. from ``U+0000`` to ``U+10FFFF``). The distinction between narrow and wide builds no longer exists and Python now behaves like a wide build, even under Windows. @@ -246,7 +246,7 @@ so ``'\U0010FFFF'[0]`` now returns ``'\U0010FFFF'`` and not ``'\uDBFF'``; * all other functions in the standard library now correctly handle - non-BMP codepoints. + non-BMP code points. * The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF`` in hexadecimal). The :c:func:`PyUnicode_GetMax` function still returns @@ -258,13 +258,13 @@ Performance and resource usage ------------------------------ -The storage of Unicode strings now depends on the highest codepoint in the string: - -* pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per codepoint; - -* BMP strings (``U+0000-U+FFFF``) use 2 bytes per codepoint; - -* non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint. +The storage of Unicode strings now depends on the highest code point in the string: + +* pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per code point; + +* BMP strings (``U+0000-U+FFFF``) use 2 bytes per code point; + +* non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per code point. The net effect is that for most applications, memory usage of string storage should decrease significantly - especially compared to former diff --git a/Lib/codecs.py b/Lib/codecs.py --- a/Lib/codecs.py +++ b/Lib/codecs.py @@ -123,7 +123,7 @@ Python will use the official U+FFFD REPLACEMENT CHARACTER for the builtin Unicode codecs on decoding and '?' on encoding. - 'surrogateescape' - replace with private codepoints U+DCnn. + 'surrogateescape' - replace with private code points U+DCnn. 'xmlcharrefreplace' - Replace with the appropriate XML character reference (only for encoding). 'backslashreplace' - Replace with backslashed escape sequences diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -273,7 +273,7 @@ bpayload = payload.encode('ascii') except UnicodeError: # This won't happen for RFC compliant messages (messages - # containing only ASCII codepoints in the unicode input). + # containing only ASCII code points in the unicode input). # If it does happen, turn the string into bytes in a way # guaranteed not to fail. bpayload = payload.encode('raw-unicode-escape') diff --git a/Lib/html/entities.py b/Lib/html/entities.py --- a/Lib/html/entities.py +++ b/Lib/html/entities.py @@ -1,6 +1,6 @@ """HTML character entity references.""" -# maps the HTML entity name to the Unicode codepoint +# maps the HTML entity name to the Unicode code point name2codepoint = { 'AElig': 0x00c6, # latin capital letter AE = latin capital ligature AE, U+00C6 ISOlat1 'Aacute': 0x00c1, # latin capital letter A with acute, U+00C1 ISOlat1 @@ -2492,7 +2492,7 @@ 'zwnj;': '\u200c', } -# maps the Unicode codepoint to the HTML entity name +# maps the Unicode code point to the HTML entity name codepoint2name = {} # maps the HTML entity name to the character diff --git a/Lib/test/multibytecodec_support.py b/Lib/test/multibytecodec_support.py --- a/Lib/test/multibytecodec_support.py +++ b/Lib/test/multibytecodec_support.py @@ -21,7 +21,7 @@ roundtriptest = 1 # set if roundtrip is possible with unicode has_iso10646 = 0 # set if this encoding contains whole iso10646 map xmlcharnametest = None # string to test xmlcharrefreplace - unmappedunicode = '\udeee' # a unicode codepoint that is not mapped. + unmappedunicode = '\udeee' # a unicode code point that is not mapped. def setUp(self): if self.codec is None: diff --git a/Lib/test/test_html.py b/Lib/test/test_html.py --- a/Lib/test/test_html.py +++ b/Lib/test/test_html.py @@ -48,10 +48,10 @@ check(s % num, char) for end in [' ', 'X']: check((s+end) % num, char+end) - # check invalid codepoints + # check invalid code points for cp in [0xD800, 0xDB00, 0xDC00, 0xDFFF, 0x110000]: check_num(cp, '\uFFFD') - # check more invalid codepoints + # check more invalid code points for cp in [0x1, 0xb, 0xe, 0x7f, 0xfffe, 0xffff, 0x10fffe, 0x10ffff]: check_num(cp, '') # check invalid numbers diff --git a/Lib/test/test_multibytecodec.py b/Lib/test/test_multibytecodec.py --- a/Lib/test/test_multibytecodec.py +++ b/Lib/test/test_multibytecodec.py @@ -80,7 +80,7 @@ self.assertEqual(encoder.reset(), None) def test_stateful(self): - # jisx0213 encoder is stateful for a few codepoints. eg) + # jisx0213 encoder is stateful for a few code points. eg) # U+00E6 => A9DC # U+00E6 U+0300 => ABC4 # U+0300 => ABDC diff --git a/Lib/test/test_stringprep.py b/Lib/test/test_stringprep.py --- a/Lib/test/test_stringprep.py +++ b/Lib/test/test_stringprep.py @@ -1,5 +1,5 @@ # To fully test this module, we would need a copy of the stringprep tables. -# Since we don't have them, this test checks only a few codepoints. +# Since we don't have them, this test checks only a few code points. import unittest from test import support diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1441,9 +1441,9 @@ def test_utf8_decode_invalid_sequences(self): # continuation bytes in a sequence of 2, 3, or 4 bytes continuation_bytes = [bytes([x]) for x in range(0x80, 0xC0)] - # start bytes of a 2-byte sequence equivalent to codepoints < 0x7F + # start bytes of a 2-byte sequence equivalent to code points < 0x7F invalid_2B_seq_start_bytes = [bytes([x]) for x in range(0xC0, 0xC2)] - # start bytes of a 4-byte sequence equivalent to codepoints > 0x10FFFF + # start bytes of a 4-byte sequence equivalent to code points > 0x10FFFF invalid_4B_seq_start_bytes = [bytes([x]) for x in range(0xF5, 0xF8)] invalid_start_bytes = ( continuation_bytes + invalid_2B_seq_start_bytes + diff --git a/Modules/cjkcodecs/_codecs_cn.c b/Modules/cjkcodecs/_codecs_cn.c --- a/Modules/cjkcodecs/_codecs_cn.c +++ b/Modules/cjkcodecs/_codecs_cn.c @@ -15,7 +15,7 @@ #undef hz #endif -/* GBK and GB2312 map differently in few codepoints that are listed below: +/* GBK and GB2312 map differently in few code points that are listed below: * * gb2312 gbk * A1A4 U+30FB KATAKANA MIDDLE DOT U+00B7 MIDDLE DOT diff --git a/Modules/cjkcodecs/_codecs_hk.c b/Modules/cjkcodecs/_codecs_hk.c --- a/Modules/cjkcodecs/_codecs_hk.c +++ b/Modules/cjkcodecs/_codecs_hk.c @@ -171,7 +171,7 @@ default: return 1; } - NEXT_IN(2); /* all decoded codepoints are pairs, above. */ + NEXT_IN(2); /* all decoded code points are pairs, above. */ } return 0; diff --git a/Modules/cjkcodecs/_codecs_kr.c b/Modules/cjkcodecs/_codecs_kr.c --- a/Modules/cjkcodecs/_codecs_kr.c +++ b/Modules/cjkcodecs/_codecs_kr.c @@ -69,7 +69,7 @@ OUTBYTE1(EUCKR_JAMO_FIRSTBYTE); OUTBYTE2(EUCKR_JAMO_FILLER); - /* All codepoints in CP949 extension are in unicode + /* All code points in CP949 extension are in unicode * Hangul Syllable area. */ assert(0xac00 <= c && c <= 0xd7a3); c -= 0xac00; diff --git a/Modules/cjkcodecs/cjkcodecs.h b/Modules/cjkcodecs/cjkcodecs.h --- a/Modules/cjkcodecs/cjkcodecs.h +++ b/Modules/cjkcodecs/cjkcodecs.h @@ -12,10 +12,10 @@ #include "multibytecodec.h" -/* a unicode "undefined" codepoint */ +/* a unicode "undefined" code point */ #define UNIINV 0xFFFE -/* internal-use DBCS codepoints which aren't used by any charsets */ +/* internal-use DBCS code points which aren't used by any charsets */ #define NOCHAR 0xFFFF #define MULTIC 0xFFFE #define DBCINV 0xFFFD diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c --- a/Modules/unicodedata.c +++ b/Modules/unicodedata.c @@ -976,7 +976,7 @@ (0x2B740 <= code && code <= 0x2B81D); /* CJK Ideograph Extension D */ } -/* macros used to determine if the given codepoint is in the PUA range that +/* macros used to determine if the given code point is in the PUA range that * we are using to store aliases and named sequences */ #define IS_ALIAS(cp) ((cp >= aliases_start) && (cp < aliases_end)) #define IS_NAMED_SEQ(cp) ((cp >= named_sequences_start) && \ @@ -986,7 +986,7 @@ _getucname(PyObject *self, Py_UCS4 code, char* buffer, int buflen, int with_alias_and_seq) { - /* Find the name associated with the given codepoint. + /* Find the name associated with the given code point. * If with_alias_and_seq is 1, check for names in the Private Use Area 15 * that we are using for aliases and named sequences. */ int offset; @@ -997,7 +997,7 @@ if (code >= 0x110000) return 0; - /* XXX should we just skip all the codepoints in the PUAs here? */ + /* XXX should we just skip all the code points in the PUAs here? */ if (!with_alias_and_seq && (IS_ALIAS(code) || IS_NAMED_SEQ(code))) return 0; @@ -1125,8 +1125,8 @@ /* check if named sequences are allowed */ if (!with_named_seq && IS_NAMED_SEQ(cp)) return 0; - /* if the codepoint is in the PUA range that we use for aliases, - * convert it to obtain the right codepoint */ + /* if the code point is in the PUA range that we use for aliases, + * convert it to obtain the right code point */ if (IS_ALIAS(cp)) *code = name_aliases[cp-aliases_start]; else @@ -1138,9 +1138,9 @@ _getcode(PyObject* self, const char* name, int namelen, Py_UCS4* code, int with_named_seq) { - /* Return the codepoint associated with the given name. + /* Return the code point associated with the given name. * Named aliases are resolved too (unless self != NULL (i.e. we are using - * 3.2.0)). If with_named_seq is 1, returns the PUA codepoint that we are + * 3.2.0)). If with_named_seq is 1, returns the PUA code point that we are * using for the named sequence, and the caller must then convert it. */ unsigned int h, v; unsigned int mask = code_size-1; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5048,7 +5048,7 @@ } if (Py_UNICODE_IS_SURROGATE(ch)) { - errmsg = "codepoint in surrogate code point range(0xd800, 0xe000)"; + errmsg = "code point in surrogate code point range(0xd800, 0xe000)"; startinpos = ((const char *)q) - starts; endinpos = startinpos + 4; } @@ -5067,7 +5067,7 @@ q += 4; continue; } - errmsg = "codepoint not in range(0x110000)"; + errmsg = "code point not in range(0x110000)"; startinpos = ((const char *)q) - starts; endinpos = startinpos + 4; } diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1368,7 +1368,7 @@ implementation -- Python implementation information.\n\ int_info -- a struct sequence with information about the int implementation.\n\ maxsize -- the largest supported length of containers.\n\ -maxunicode -- the value of the largest Unicode codepoint\n\ +maxunicode -- the value of the largest Unicode code point\n\ platform -- platform identifier\n\ prefix -- prefix used to find the Python library\n\ thread_info -- a struct sequence with information about the thread implementation.\n\ diff --git a/Tools/unicode/gencodec.py b/Tools/unicode/gencodec.py --- a/Tools/unicode/gencodec.py +++ b/Tools/unicode/gencodec.py @@ -34,7 +34,7 @@ # Standard undefined Unicode code point UNI_UNDEFINED = chr(0xFFFE) -# Placeholder for a missing codepoint +# Placeholder for a missing code point MISSING_CODE = -1 mapRE = re.compile('((?:0x[0-9a-fA-F]+\+?)+)' -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 16:23:05 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 18 Jan 2015 15:23:05 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323266=3A_Much_fas?= =?utf-8?q?ter_implementation_of_ipaddress=2Ecollapse=5Faddresses=28=29_wh?= =?utf-8?q?en?= Message-ID: <20150118152253.118096.88116@psf.io> https://hg.python.org/cpython/rev/f7508a176a09 changeset: 94208:f7508a176a09 parent: 94206:c79abee84a39 user: Antoine Pitrou date: Sun Jan 18 16:22:47 2015 +0100 summary: Issue #23266: Much faster implementation of ipaddress.collapse_addresses() when there are many non-consecutive addresses. files: Lib/ipaddress.py | 18 +++++++++++------- Lib/test/test_ipaddress.py | 3 ++- Misc/NEWS | 3 +++ 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -170,16 +170,19 @@ addresses: a list of IPv#Address objects. Returns: - A tuple containing the first and last IP addresses in the sequence. + A tuple containing the first and last IP addresses in the sequence, + and the number of distinct IP addresses in the sequence. """ first = last = addresses[0] + i = 1 for ip in addresses[1:]: if ip._ip == last._ip + 1: last = ip + i += 1 else: break - return (first, last) + return (first, last, i) def _count_righthand_zero_bits(number, bits): @@ -346,12 +349,13 @@ ip, nets[-1])) nets.append(ip) - # sort and dedup - ips = sorted(set(ips)) - + # sort + ips = sorted(ips) + + # find consecutive address ranges in the sorted sequence and summarize them while i < len(ips): - (first, last) = _find_address_range(ips[i:]) - i = ips.index(last) + 1 + (first, last, items) = _find_address_range(ips[i:]) + i = items + i addrs.extend(summarize_address_range(first, last)) return _collapse_addresses_internal(addrs + nets) diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -766,10 +766,11 @@ 2 ** ipaddress.IPV6LENGTH) def testInternals(self): - first, last = ipaddress._find_address_range([ + first, last, nitems = ipaddress._find_address_range([ ipaddress.IPv4Address('10.10.10.10'), ipaddress.IPv4Address('10.10.10.12')]) self.assertEqual(first, last) + self.assertEqual(nitems, 1) self.assertEqual(128, ipaddress._count_righthand_zero_bits(0, 128)) self.assertEqual("IPv4Network('1.2.3.0/24')", repr(self.ipv4_network)) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,9 @@ Library ------- +- Issue #23266: Much faster implementation of ipaddress.collapse_addresses() + when there are many non-consecutive addresses. + - Issue #23098: 64-bit dev_t is now supported in the os module. - Issue #21817: When an exception is raised in a task submitted to a -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 17:42:32 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 18 Jan 2015 16:42:32 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323248=3A_Update_ssl_error_codes_from_latest_Ope?= =?utf-8?q?nSSL_git_master=2E?= Message-ID: <20150118164232.104130.20353@psf.io> https://hg.python.org/cpython/rev/15f46b850257 changeset: 94210:15f46b850257 parent: 94208:f7508a176a09 parent: 94209:2eea364c2863 user: Antoine Pitrou date: Sun Jan 18 17:40:17 2015 +0100 summary: Issue #23248: Update ssl error codes from latest OpenSSL git master. files: Misc/NEWS | 2 + Modules/_ssl_data.h | 297 ++++++++++++++++++++++++- Tools/ssl/make_ssl_data.py | 57 +++- 3 files changed, 336 insertions(+), 20 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,8 @@ Library ------- +- Issue #23248: Update ssl error codes from latest OpenSSL git master. + - Issue #23266: Much faster implementation of ipaddress.collapse_addresses() when there are many non-consecutive addresses. diff --git a/Modules/_ssl_data.h b/Modules/_ssl_data.h --- a/Modules/_ssl_data.h +++ b/Modules/_ssl_data.h @@ -1,5 +1,5 @@ /* File generated by Tools/ssl/make_ssl_data.py */ -/* Generated on 2012-05-16T23:56:40.981382 */ +/* Generated on 2015-01-17T20:33:43.377453 */ static struct py_ssl_library_code library_codes[] = { {"PEM", ERR_LIB_PEM}, @@ -179,6 +179,11 @@ #else {"BAD_CHECKSUM", ERR_LIB_SSL, 104}, #endif + #ifdef SSL_R_BAD_DATA + {"BAD_DATA", ERR_LIB_SSL, SSL_R_BAD_DATA}, + #else + {"BAD_DATA", ERR_LIB_SSL, 390}, + #endif #ifdef SSL_R_BAD_DATA_RETURNED_BY_CALLBACK {"BAD_DATA_RETURNED_BY_CALLBACK", ERR_LIB_SSL, SSL_R_BAD_DATA_RETURNED_BY_CALLBACK}, #else @@ -309,6 +314,46 @@ #else {"BAD_SIGNATURE", ERR_LIB_SSL, 123}, #endif + #ifdef SSL_R_BAD_SRP_A_LENGTH + {"BAD_SRP_A_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_A_LENGTH}, + #else + {"BAD_SRP_A_LENGTH", ERR_LIB_SSL, 347}, + #endif + #ifdef SSL_R_BAD_SRP_B_LENGTH + {"BAD_SRP_B_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_B_LENGTH}, + #else + {"BAD_SRP_B_LENGTH", ERR_LIB_SSL, 348}, + #endif + #ifdef SSL_R_BAD_SRP_G_LENGTH + {"BAD_SRP_G_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_G_LENGTH}, + #else + {"BAD_SRP_G_LENGTH", ERR_LIB_SSL, 349}, + #endif + #ifdef SSL_R_BAD_SRP_N_LENGTH + {"BAD_SRP_N_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_N_LENGTH}, + #else + {"BAD_SRP_N_LENGTH", ERR_LIB_SSL, 350}, + #endif + #ifdef SSL_R_BAD_SRP_PARAMETERS + {"BAD_SRP_PARAMETERS", ERR_LIB_SSL, SSL_R_BAD_SRP_PARAMETERS}, + #else + {"BAD_SRP_PARAMETERS", ERR_LIB_SSL, 371}, + #endif + #ifdef SSL_R_BAD_SRP_S_LENGTH + {"BAD_SRP_S_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_S_LENGTH}, + #else + {"BAD_SRP_S_LENGTH", ERR_LIB_SSL, 351}, + #endif + #ifdef SSL_R_BAD_SRTP_MKI_VALUE + {"BAD_SRTP_MKI_VALUE", ERR_LIB_SSL, SSL_R_BAD_SRTP_MKI_VALUE}, + #else + {"BAD_SRTP_MKI_VALUE", ERR_LIB_SSL, 352}, + #endif + #ifdef SSL_R_BAD_SRTP_PROTECTION_PROFILE_LIST + {"BAD_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, SSL_R_BAD_SRTP_PROTECTION_PROFILE_LIST}, + #else + {"BAD_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, 353}, + #endif #ifdef SSL_R_BAD_SSL_FILETYPE {"BAD_SSL_FILETYPE", ERR_LIB_SSL, SSL_R_BAD_SSL_FILETYPE}, #else @@ -324,6 +369,11 @@ #else {"BAD_STATE", ERR_LIB_SSL, 126}, #endif + #ifdef SSL_R_BAD_VALUE + {"BAD_VALUE", ERR_LIB_SSL, SSL_R_BAD_VALUE}, + #else + {"BAD_VALUE", ERR_LIB_SSL, 384}, + #endif #ifdef SSL_R_BAD_WRITE_RETRY {"BAD_WRITE_RETRY", ERR_LIB_SSL, SSL_R_BAD_WRITE_RETRY}, #else @@ -354,6 +404,16 @@ #else {"CA_DN_TOO_LONG", ERR_LIB_SSL, 132}, #endif + #ifdef SSL_R_CA_KEY_TOO_SMALL + {"CA_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_CA_KEY_TOO_SMALL}, + #else + {"CA_KEY_TOO_SMALL", ERR_LIB_SSL, 397}, + #endif + #ifdef SSL_R_CA_MD_TOO_WEAK + {"CA_MD_TOO_WEAK", ERR_LIB_SSL, SSL_R_CA_MD_TOO_WEAK}, + #else + {"CA_MD_TOO_WEAK", ERR_LIB_SSL, 398}, + #endif #ifdef SSL_R_CCS_RECEIVED_EARLY {"CCS_RECEIVED_EARLY", ERR_LIB_SSL, SSL_R_CCS_RECEIVED_EARLY}, #else @@ -364,6 +424,11 @@ #else {"CERTIFICATE_VERIFY_FAILED", ERR_LIB_SSL, 134}, #endif + #ifdef SSL_R_CERT_CB_ERROR + {"CERT_CB_ERROR", ERR_LIB_SSL, SSL_R_CERT_CB_ERROR}, + #else + {"CERT_CB_ERROR", ERR_LIB_SSL, 377}, + #endif #ifdef SSL_R_CERT_LENGTH_MISMATCH {"CERT_LENGTH_MISMATCH", ERR_LIB_SSL, SSL_R_CERT_LENGTH_MISMATCH}, #else @@ -454,6 +519,11 @@ #else {"DECRYPTION_FAILED_OR_BAD_RECORD_MAC", ERR_LIB_SSL, 281}, #endif + #ifdef SSL_R_DH_KEY_TOO_SMALL + {"DH_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_DH_KEY_TOO_SMALL}, + #else + {"DH_KEY_TOO_SMALL", ERR_LIB_SSL, 394}, + #endif #ifdef SSL_R_DH_PUBLIC_VALUE_LENGTH_IS_WRONG {"DH_PUBLIC_VALUE_LENGTH_IS_WRONG", ERR_LIB_SSL, SSL_R_DH_PUBLIC_VALUE_LENGTH_IS_WRONG}, #else @@ -494,11 +564,26 @@ #else {"ECC_CERT_SHOULD_HAVE_SHA1_SIGNATURE", ERR_LIB_SSL, 323}, #endif + #ifdef SSL_R_ECDH_REQUIRED_FOR_SUITEB_MODE + {"ECDH_REQUIRED_FOR_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ECDH_REQUIRED_FOR_SUITEB_MODE}, + #else + {"ECDH_REQUIRED_FOR_SUITEB_MODE", ERR_LIB_SSL, 374}, + #endif #ifdef SSL_R_ECGROUP_TOO_LARGE_FOR_CIPHER {"ECGROUP_TOO_LARGE_FOR_CIPHER", ERR_LIB_SSL, SSL_R_ECGROUP_TOO_LARGE_FOR_CIPHER}, #else {"ECGROUP_TOO_LARGE_FOR_CIPHER", ERR_LIB_SSL, 310}, #endif + #ifdef SSL_R_EE_KEY_TOO_SMALL + {"EE_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_EE_KEY_TOO_SMALL}, + #else + {"EE_KEY_TOO_SMALL", ERR_LIB_SSL, 399}, + #endif + #ifdef SSL_R_EMPTY_SRTP_PROTECTION_PROFILE_LIST + {"EMPTY_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, SSL_R_EMPTY_SRTP_PROTECTION_PROFILE_LIST}, + #else + {"EMPTY_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, 354}, + #endif #ifdef SSL_R_ENCRYPTED_LENGTH_TOO_LONG {"ENCRYPTED_LENGTH_TOO_LONG", ERR_LIB_SSL, SSL_R_ENCRYPTED_LENGTH_TOO_LONG}, #else @@ -529,6 +614,16 @@ #else {"GOT_A_FIN_BEFORE_A_CCS", ERR_LIB_SSL, 154}, #endif + #ifdef SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS + {"GOT_NEXT_PROTO_BEFORE_A_CCS", ERR_LIB_SSL, SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS}, + #else + {"GOT_NEXT_PROTO_BEFORE_A_CCS", ERR_LIB_SSL, 355}, + #endif + #ifdef SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION + {"GOT_NEXT_PROTO_WITHOUT_EXTENSION", ERR_LIB_SSL, SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION}, + #else + {"GOT_NEXT_PROTO_WITHOUT_EXTENSION", ERR_LIB_SSL, 356}, + #endif #ifdef SSL_R_HTTPS_PROXY_REQUEST {"HTTPS_PROXY_REQUEST", ERR_LIB_SSL, SSL_R_HTTPS_PROXY_REQUEST}, #else @@ -544,6 +639,16 @@ #else {"ILLEGAL_PADDING", ERR_LIB_SSL, 283}, #endif + #ifdef SSL_R_ILLEGAL_SUITEB_DIGEST + {"ILLEGAL_SUITEB_DIGEST", ERR_LIB_SSL, SSL_R_ILLEGAL_SUITEB_DIGEST}, + #else + {"ILLEGAL_SUITEB_DIGEST", ERR_LIB_SSL, 380}, + #endif + #ifdef SSL_R_INAPPROPRIATE_FALLBACK + {"INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, SSL_R_INAPPROPRIATE_FALLBACK}, + #else + {"INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, 373}, + #endif #ifdef SSL_R_INCONSISTENT_COMPRESSION {"INCONSISTENT_COMPRESSION", ERR_LIB_SSL, SSL_R_INCONSISTENT_COMPRESSION}, #else @@ -564,11 +669,26 @@ #else {"INVALID_COMPRESSION_ALGORITHM", ERR_LIB_SSL, 341}, #endif + #ifdef SSL_R_INVALID_NULL_CMD_NAME + {"INVALID_NULL_CMD_NAME", ERR_LIB_SSL, SSL_R_INVALID_NULL_CMD_NAME}, + #else + {"INVALID_NULL_CMD_NAME", ERR_LIB_SSL, 385}, + #endif #ifdef SSL_R_INVALID_PURPOSE {"INVALID_PURPOSE", ERR_LIB_SSL, SSL_R_INVALID_PURPOSE}, #else {"INVALID_PURPOSE", ERR_LIB_SSL, 278}, #endif + #ifdef SSL_R_INVALID_SERVERINFO_DATA + {"INVALID_SERVERINFO_DATA", ERR_LIB_SSL, SSL_R_INVALID_SERVERINFO_DATA}, + #else + {"INVALID_SERVERINFO_DATA", ERR_LIB_SSL, 388}, + #endif + #ifdef SSL_R_INVALID_SRP_USERNAME + {"INVALID_SRP_USERNAME", ERR_LIB_SSL, SSL_R_INVALID_SRP_USERNAME}, + #else + {"INVALID_SRP_USERNAME", ERR_LIB_SSL, 357}, + #endif #ifdef SSL_R_INVALID_STATUS_RESPONSE {"INVALID_STATUS_RESPONSE", ERR_LIB_SSL, SSL_R_INVALID_STATUS_RESPONSE}, #else @@ -689,6 +809,16 @@ #else {"MISSING_DSA_SIGNING_CERT", ERR_LIB_SSL, 165}, #endif + #ifdef SSL_R_MISSING_ECDH_CERT + {"MISSING_ECDH_CERT", ERR_LIB_SSL, SSL_R_MISSING_ECDH_CERT}, + #else + {"MISSING_ECDH_CERT", ERR_LIB_SSL, 382}, + #endif + #ifdef SSL_R_MISSING_ECDSA_SIGNING_CERT + {"MISSING_ECDSA_SIGNING_CERT", ERR_LIB_SSL, SSL_R_MISSING_ECDSA_SIGNING_CERT}, + #else + {"MISSING_ECDSA_SIGNING_CERT", ERR_LIB_SSL, 381}, + #endif #ifdef SSL_R_MISSING_EXPORT_TMP_DH_KEY {"MISSING_EXPORT_TMP_DH_KEY", ERR_LIB_SSL, SSL_R_MISSING_EXPORT_TMP_DH_KEY}, #else @@ -714,6 +844,11 @@ #else {"MISSING_RSA_SIGNING_CERT", ERR_LIB_SSL, 170}, #endif + #ifdef SSL_R_MISSING_SRP_PARAM + {"MISSING_SRP_PARAM", ERR_LIB_SSL, SSL_R_MISSING_SRP_PARAM}, + #else + {"MISSING_SRP_PARAM", ERR_LIB_SSL, 358}, + #endif #ifdef SSL_R_MISSING_TMP_DH_KEY {"MISSING_TMP_DH_KEY", ERR_LIB_SSL, SSL_R_MISSING_TMP_DH_KEY}, #else @@ -739,6 +874,11 @@ #else {"MISSING_VERIFY_MESSAGE", ERR_LIB_SSL, 174}, #endif + #ifdef SSL_R_MULTIPLE_SGC_RESTARTS + {"MULTIPLE_SGC_RESTARTS", ERR_LIB_SSL, SSL_R_MULTIPLE_SGC_RESTARTS}, + #else + {"MULTIPLE_SGC_RESTARTS", ERR_LIB_SSL, 346}, + #endif #ifdef SSL_R_NON_SSLV2_INITIAL_PACKET {"NON_SSLV2_INITIAL_PACKET", ERR_LIB_SSL, SSL_R_NON_SSLV2_INITIAL_PACKET}, #else @@ -819,6 +959,11 @@ #else {"NO_METHOD_SPECIFIED", ERR_LIB_SSL, 188}, #endif + #ifdef SSL_R_NO_PEM_EXTENSIONS + {"NO_PEM_EXTENSIONS", ERR_LIB_SSL, SSL_R_NO_PEM_EXTENSIONS}, + #else + {"NO_PEM_EXTENSIONS", ERR_LIB_SSL, 389}, + #endif #ifdef SSL_R_NO_PRIVATEKEY {"NO_PRIVATEKEY", ERR_LIB_SSL, SSL_R_NO_PRIVATEKEY}, #else @@ -854,6 +999,16 @@ #else {"NO_SHARED_CIPHER", ERR_LIB_SSL, 193}, #endif + #ifdef SSL_R_NO_SHARED_SIGATURE_ALGORITHMS + {"NO_SHARED_SIGATURE_ALGORITHMS", ERR_LIB_SSL, SSL_R_NO_SHARED_SIGATURE_ALGORITHMS}, + #else + {"NO_SHARED_SIGATURE_ALGORITHMS", ERR_LIB_SSL, 376}, + #endif + #ifdef SSL_R_NO_SRTP_PROFILES + {"NO_SRTP_PROFILES", ERR_LIB_SSL, SSL_R_NO_SRTP_PROFILES}, + #else + {"NO_SRTP_PROFILES", ERR_LIB_SSL, 359}, + #endif #ifdef SSL_R_NO_VERIFY_CALLBACK {"NO_VERIFY_CALLBACK", ERR_LIB_SSL, SSL_R_NO_VERIFY_CALLBACK}, #else @@ -879,6 +1034,16 @@ #else {"OLD_SESSION_COMPRESSION_ALGORITHM_NOT_RETURNED", ERR_LIB_SSL, 344}, #endif + #ifdef SSL_R_ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE + {"ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE}, + #else + {"ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, 387}, + #endif + #ifdef SSL_R_ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE + {"ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE}, + #else + {"ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, 379}, + #endif #ifdef SSL_R_ONLY_TLS_ALLOWED_IN_FIPS_MODE {"ONLY_TLS_ALLOWED_IN_FIPS_MODE", ERR_LIB_SSL, SSL_R_ONLY_TLS_ALLOWED_IN_FIPS_MODE}, #else @@ -934,6 +1099,16 @@ #else {"PEER_ERROR_UNSUPPORTED_CERTIFICATE_TYPE", ERR_LIB_SSL, 204}, #endif + #ifdef SSL_R_PEM_NAME_BAD_PREFIX + {"PEM_NAME_BAD_PREFIX", ERR_LIB_SSL, SSL_R_PEM_NAME_BAD_PREFIX}, + #else + {"PEM_NAME_BAD_PREFIX", ERR_LIB_SSL, 391}, + #endif + #ifdef SSL_R_PEM_NAME_TOO_SHORT + {"PEM_NAME_TOO_SHORT", ERR_LIB_SSL, SSL_R_PEM_NAME_TOO_SHORT}, + #else + {"PEM_NAME_TOO_SHORT", ERR_LIB_SSL, 392}, + #endif #ifdef SSL_R_PRE_MAC_LENGTH_TOO_LONG {"PRE_MAC_LENGTH_TOO_LONG", ERR_LIB_SSL, SSL_R_PRE_MAC_LENGTH_TOO_LONG}, #else @@ -1069,11 +1244,36 @@ #else {"SHORT_READ", ERR_LIB_SSL, 219}, #endif + #ifdef SSL_R_SIGNATURE_ALGORITHMS_ERROR + {"SIGNATURE_ALGORITHMS_ERROR", ERR_LIB_SSL, SSL_R_SIGNATURE_ALGORITHMS_ERROR}, + #else + {"SIGNATURE_ALGORITHMS_ERROR", ERR_LIB_SSL, 360}, + #endif #ifdef SSL_R_SIGNATURE_FOR_NON_SIGNING_CERTIFICATE {"SIGNATURE_FOR_NON_SIGNING_CERTIFICATE", ERR_LIB_SSL, SSL_R_SIGNATURE_FOR_NON_SIGNING_CERTIFICATE}, #else {"SIGNATURE_FOR_NON_SIGNING_CERTIFICATE", ERR_LIB_SSL, 220}, #endif + #ifdef SSL_R_SRP_A_CALC + {"SRP_A_CALC", ERR_LIB_SSL, SSL_R_SRP_A_CALC}, + #else + {"SRP_A_CALC", ERR_LIB_SSL, 361}, + #endif + #ifdef SSL_R_SRTP_COULD_NOT_ALLOCATE_PROFILES + {"SRTP_COULD_NOT_ALLOCATE_PROFILES", ERR_LIB_SSL, SSL_R_SRTP_COULD_NOT_ALLOCATE_PROFILES}, + #else + {"SRTP_COULD_NOT_ALLOCATE_PROFILES", ERR_LIB_SSL, 362}, + #endif + #ifdef SSL_R_SRTP_PROTECTION_PROFILE_LIST_TOO_LONG + {"SRTP_PROTECTION_PROFILE_LIST_TOO_LONG", ERR_LIB_SSL, SSL_R_SRTP_PROTECTION_PROFILE_LIST_TOO_LONG}, + #else + {"SRTP_PROTECTION_PROFILE_LIST_TOO_LONG", ERR_LIB_SSL, 363}, + #endif + #ifdef SSL_R_SRTP_UNKNOWN_PROTECTION_PROFILE + {"SRTP_UNKNOWN_PROTECTION_PROFILE", ERR_LIB_SSL, SSL_R_SRTP_UNKNOWN_PROTECTION_PROFILE}, + #else + {"SRTP_UNKNOWN_PROTECTION_PROFILE", ERR_LIB_SSL, 364}, + #endif #ifdef SSL_R_SSL23_DOING_SESSION_ID_REUSE {"SSL23_DOING_SESSION_ID_REUSE", ERR_LIB_SSL, SSL_R_SSL23_DOING_SESSION_ID_REUSE}, #else @@ -1179,6 +1379,11 @@ #else {"SSL_LIBRARY_HAS_NO_CIPHERS", ERR_LIB_SSL, 230}, #endif + #ifdef SSL_R_SSL_NEGATIVE_LENGTH + {"SSL_NEGATIVE_LENGTH", ERR_LIB_SSL, SSL_R_SSL_NEGATIVE_LENGTH}, + #else + {"SSL_NEGATIVE_LENGTH", ERR_LIB_SSL, 372}, + #endif #ifdef SSL_R_SSL_SESSION_ID_CALLBACK_FAILED {"SSL_SESSION_ID_CALLBACK_FAILED", ERR_LIB_SSL, SSL_R_SSL_SESSION_ID_CALLBACK_FAILED}, #else @@ -1229,6 +1434,11 @@ #else {"TLSV1_ALERT_EXPORT_RESTRICTION", ERR_LIB_SSL, 1060}, #endif + #ifdef SSL_R_TLSV1_ALERT_INAPPROPRIATE_FALLBACK + {"TLSV1_ALERT_INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, SSL_R_TLSV1_ALERT_INAPPROPRIATE_FALLBACK}, + #else + {"TLSV1_ALERT_INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, 1086}, + #endif #ifdef SSL_R_TLSV1_ALERT_INSUFFICIENT_SECURITY {"TLSV1_ALERT_INSUFFICIENT_SECURITY", ERR_LIB_SSL, SSL_R_TLSV1_ALERT_INSUFFICIENT_SECURITY}, #else @@ -1294,6 +1504,21 @@ #else {"TLS_CLIENT_CERT_REQ_WITH_ANON_CIPHER", ERR_LIB_SSL, 232}, #endif + #ifdef SSL_R_TLS_HEARTBEAT_PEER_DOESNT_ACCEPT + {"TLS_HEARTBEAT_PEER_DOESNT_ACCEPT", ERR_LIB_SSL, SSL_R_TLS_HEARTBEAT_PEER_DOESNT_ACCEPT}, + #else + {"TLS_HEARTBEAT_PEER_DOESNT_ACCEPT", ERR_LIB_SSL, 365}, + #endif + #ifdef SSL_R_TLS_HEARTBEAT_PENDING + {"TLS_HEARTBEAT_PENDING", ERR_LIB_SSL, SSL_R_TLS_HEARTBEAT_PENDING}, + #else + {"TLS_HEARTBEAT_PENDING", ERR_LIB_SSL, 366}, + #endif + #ifdef SSL_R_TLS_ILLEGAL_EXPORTER_LABEL + {"TLS_ILLEGAL_EXPORTER_LABEL", ERR_LIB_SSL, SSL_R_TLS_ILLEGAL_EXPORTER_LABEL}, + #else + {"TLS_ILLEGAL_EXPORTER_LABEL", ERR_LIB_SSL, 367}, + #endif #ifdef SSL_R_TLS_INVALID_ECPOINTFORMAT_LIST {"TLS_INVALID_ECPOINTFORMAT_LIST", ERR_LIB_SSL, SSL_R_TLS_INVALID_ECPOINTFORMAT_LIST}, #else @@ -1399,6 +1624,16 @@ #else {"UNKNOWN_CIPHER_TYPE", ERR_LIB_SSL, 249}, #endif + #ifdef SSL_R_UNKNOWN_CMD_NAME + {"UNKNOWN_CMD_NAME", ERR_LIB_SSL, SSL_R_UNKNOWN_CMD_NAME}, + #else + {"UNKNOWN_CMD_NAME", ERR_LIB_SSL, 386}, + #endif + #ifdef SSL_R_UNKNOWN_DIGEST + {"UNKNOWN_DIGEST", ERR_LIB_SSL, SSL_R_UNKNOWN_DIGEST}, + #else + {"UNKNOWN_DIGEST", ERR_LIB_SSL, 368}, + #endif #ifdef SSL_R_UNKNOWN_KEY_EXCHANGE_TYPE {"UNKNOWN_KEY_EXCHANGE_TYPE", ERR_LIB_SSL, SSL_R_UNKNOWN_KEY_EXCHANGE_TYPE}, #else @@ -1469,16 +1704,36 @@ #else {"UNSUPPORTED_STATUS_TYPE", ERR_LIB_SSL, 329}, #endif + #ifdef SSL_R_USE_SRTP_NOT_NEGOTIATED + {"USE_SRTP_NOT_NEGOTIATED", ERR_LIB_SSL, SSL_R_USE_SRTP_NOT_NEGOTIATED}, + #else + {"USE_SRTP_NOT_NEGOTIATED", ERR_LIB_SSL, 369}, + #endif + #ifdef SSL_R_VERSION_TOO_LOW + {"VERSION_TOO_LOW", ERR_LIB_SSL, SSL_R_VERSION_TOO_LOW}, + #else + {"VERSION_TOO_LOW", ERR_LIB_SSL, 396}, + #endif #ifdef SSL_R_WRITE_BIO_NOT_SET {"WRITE_BIO_NOT_SET", ERR_LIB_SSL, SSL_R_WRITE_BIO_NOT_SET}, #else {"WRITE_BIO_NOT_SET", ERR_LIB_SSL, 260}, #endif + #ifdef SSL_R_WRONG_CERTIFICATE_TYPE + {"WRONG_CERTIFICATE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_CERTIFICATE_TYPE}, + #else + {"WRONG_CERTIFICATE_TYPE", ERR_LIB_SSL, 383}, + #endif #ifdef SSL_R_WRONG_CIPHER_RETURNED {"WRONG_CIPHER_RETURNED", ERR_LIB_SSL, SSL_R_WRONG_CIPHER_RETURNED}, #else {"WRONG_CIPHER_RETURNED", ERR_LIB_SSL, 261}, #endif + #ifdef SSL_R_WRONG_CURVE + {"WRONG_CURVE", ERR_LIB_SSL, SSL_R_WRONG_CURVE}, + #else + {"WRONG_CURVE", ERR_LIB_SSL, 378}, + #endif #ifdef SSL_R_WRONG_MESSAGE_TYPE {"WRONG_MESSAGE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_MESSAGE_TYPE}, #else @@ -1499,6 +1754,11 @@ #else {"WRONG_SIGNATURE_SIZE", ERR_LIB_SSL, 265}, #endif + #ifdef SSL_R_WRONG_SIGNATURE_TYPE + {"WRONG_SIGNATURE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_SIGNATURE_TYPE}, + #else + {"WRONG_SIGNATURE_TYPE", ERR_LIB_SSL, 370}, + #endif #ifdef SSL_R_WRONG_SSL_VERSION {"WRONG_SSL_VERSION", ERR_LIB_SSL, SSL_R_WRONG_SSL_VERSION}, #else @@ -1519,6 +1779,11 @@ #else {"X509_VERIFICATION_SETUP_PROBLEMS", ERR_LIB_SSL, 269}, #endif + #ifdef X509_R_AKID_MISMATCH + {"AKID_MISMATCH", ERR_LIB_X509, X509_R_AKID_MISMATCH}, + #else + {"AKID_MISMATCH", ERR_LIB_X509, 110}, + #endif #ifdef X509_R_BAD_X509_FILETYPE {"BAD_X509_FILETYPE", ERR_LIB_X509, X509_R_BAD_X509_FILETYPE}, #else @@ -1539,11 +1804,26 @@ #else {"CERT_ALREADY_IN_HASH_TABLE", ERR_LIB_X509, 101}, #endif + #ifdef X509_R_CRL_ALREADY_DELTA + {"CRL_ALREADY_DELTA", ERR_LIB_X509, X509_R_CRL_ALREADY_DELTA}, + #else + {"CRL_ALREADY_DELTA", ERR_LIB_X509, 127}, + #endif + #ifdef X509_R_CRL_VERIFY_FAILURE + {"CRL_VERIFY_FAILURE", ERR_LIB_X509, X509_R_CRL_VERIFY_FAILURE}, + #else + {"CRL_VERIFY_FAILURE", ERR_LIB_X509, 131}, + #endif #ifdef X509_R_ERR_ASN1_LIB {"ERR_ASN1_LIB", ERR_LIB_X509, X509_R_ERR_ASN1_LIB}, #else {"ERR_ASN1_LIB", ERR_LIB_X509, 102}, #endif + #ifdef X509_R_IDP_MISMATCH + {"IDP_MISMATCH", ERR_LIB_X509, X509_R_IDP_MISMATCH}, + #else + {"IDP_MISMATCH", ERR_LIB_X509, 128}, + #endif #ifdef X509_R_INVALID_DIRECTORY {"INVALID_DIRECTORY", ERR_LIB_X509, X509_R_INVALID_DIRECTORY}, #else @@ -1559,6 +1839,11 @@ #else {"INVALID_TRUST", ERR_LIB_X509, 123}, #endif + #ifdef X509_R_ISSUER_MISMATCH + {"ISSUER_MISMATCH", ERR_LIB_X509, X509_R_ISSUER_MISMATCH}, + #else + {"ISSUER_MISMATCH", ERR_LIB_X509, 129}, + #endif #ifdef X509_R_KEY_TYPE_MISMATCH {"KEY_TYPE_MISMATCH", ERR_LIB_X509, X509_R_KEY_TYPE_MISMATCH}, #else @@ -1584,11 +1869,21 @@ #else {"METHOD_NOT_SUPPORTED", ERR_LIB_X509, 124}, #endif + #ifdef X509_R_NEWER_CRL_NOT_NEWER + {"NEWER_CRL_NOT_NEWER", ERR_LIB_X509, X509_R_NEWER_CRL_NOT_NEWER}, + #else + {"NEWER_CRL_NOT_NEWER", ERR_LIB_X509, 132}, + #endif #ifdef X509_R_NO_CERT_SET_FOR_US_TO_VERIFY {"NO_CERT_SET_FOR_US_TO_VERIFY", ERR_LIB_X509, X509_R_NO_CERT_SET_FOR_US_TO_VERIFY}, #else {"NO_CERT_SET_FOR_US_TO_VERIFY", ERR_LIB_X509, 105}, #endif + #ifdef X509_R_NO_CRL_NUMBER + {"NO_CRL_NUMBER", ERR_LIB_X509, X509_R_NO_CRL_NUMBER}, + #else + {"NO_CRL_NUMBER", ERR_LIB_X509, 130}, + #endif #ifdef X509_R_PUBLIC_KEY_DECODE_ERROR {"PUBLIC_KEY_DECODE_ERROR", ERR_LIB_X509, X509_R_PUBLIC_KEY_DECODE_ERROR}, #else diff --git a/Tools/ssl/make_ssl_data.py b/Tools/ssl/make_ssl_data.py --- a/Tools/ssl/make_ssl_data.py +++ b/Tools/ssl/make_ssl_data.py @@ -5,8 +5,7 @@ `library` and `reason` mnemnonics to a more recent OpenSSL version. It takes two arguments: -- the path to the OpenSSL include files' directory - (e.g. openssl-1.0.1-beta3/include/openssl/) +- the path to the OpenSSL source tree (e.g. git checkout) - the path to the C file to be generated (probably Modules/_ssl_data.h) """ @@ -15,9 +14,10 @@ import os import re import sys +import _ssl -def parse_error_codes(h_file, prefix): +def parse_error_codes(h_file, prefix, libcode): pat = re.compile(r"#define\W+(%s([\w]+))\W+(\d+)\b" % re.escape(prefix)) codes = [] with open(h_file, "r", encoding="latin1") as f: @@ -26,7 +26,8 @@ if match: code, name, num = match.groups() num = int(num) - codes.append((code, name, num)) + # e.g. ("SSL_R_BAD_DATA", ("ERR_LIB_SSL", "BAD_DATA", 390)) + codes.append((code, (libcode, name, num))) return codes if __name__ == "__main__": @@ -34,12 +35,32 @@ outfile = sys.argv[2] use_stdout = outfile == '-' f = sys.stdout if use_stdout else open(outfile, "w") - error_libraries = ( - # (library code, mnemonic, error prefix, header file) - ('ERR_LIB_PEM', 'PEM', 'PEM_R_', 'pem.h'), - ('ERR_LIB_SSL', 'SSL', 'SSL_R_', 'ssl.h'), - ('ERR_LIB_X509', 'X509', 'X509_R_', 'x509.h'), - ) + error_libraries = { + # mnemonic -> (library code, error prefix, header file) + 'PEM': ('ERR_LIB_PEM', 'PEM_R_', 'crypto/pem/pem.h'), + 'SSL': ('ERR_LIB_SSL', 'SSL_R_', 'ssl/ssl.h'), + 'X509': ('ERR_LIB_X509', 'X509_R_', 'crypto/x509/x509.h'), + } + + # Read codes from libraries + new_codes = [] + for libcode, prefix, h_file in sorted(error_libraries.values()): + new_codes += parse_error_codes(os.path.join(openssl_inc, h_file), + prefix, libcode) + new_code_nums = set((libcode, num) + for (code, (libcode, name, num)) in new_codes) + + # Merge with existing codes (in case some old codes disappeared). + codes = {} + for errname, (libnum, errnum) in _ssl.err_names_to_codes.items(): + lib = error_libraries[_ssl.lib_codes_to_names[libnum]] + libcode = lib[0] # e.g. ERR_LIB_PEM + errcode = lib[1] + errname # e.g. SSL_R_BAD_SSL_SESSION_ID_LENGTH + # Only keep it if the numeric codes weren't reused + if (libcode, errnum) not in new_code_nums: + codes[errcode] = libcode, errname, errnum + codes.update(dict(new_codes)) + def w(l): f.write(l + "\n") w("/* File generated by Tools/ssl/make_ssl_data.py */") @@ -47,21 +68,19 @@ w("") w("static struct py_ssl_library_code library_codes[] = {") - for libcode, mnemo, _, _ in error_libraries: + for mnemo, (libcode, _, _) in sorted(error_libraries.items()): w(' {"%s", %s},' % (mnemo, libcode)) w(' { NULL }') w('};') w("") w("static struct py_ssl_error_code error_codes[] = {") - for libcode, _, prefix, h_file in error_libraries: - codes = parse_error_codes(os.path.join(openssl_inc, h_file), prefix) - for code, name, num in sorted(codes): - w(' #ifdef %s' % (code)) - w(' {"%s", %s, %s},' % (name, libcode, code)) - w(' #else') - w(' {"%s", %s, %d},' % (name, libcode, num)) - w(' #endif') + for errcode, (libcode, name, num) in sorted(codes.items()): + w(' #ifdef %s' % (errcode)) + w(' {"%s", %s, %s},' % (name, libcode, errcode)) + w(' #else') + w(' {"%s", %s, %d},' % (name, libcode, num)) + w(' #endif') w(' { NULL }') w('};') if not use_stdout: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 17:42:32 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 18 Jan 2015 16:42:32 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMjQ4?= =?utf-8?q?=3A_Update_ssl_error_codes_from_latest_OpenSSL_git_master=2E?= Message-ID: <20150118164232.93187.54217@psf.io> https://hg.python.org/cpython/rev/6e03c18f5447 changeset: 94211:6e03c18f5447 branch: 2.7 parent: 94207:2db41d551a4f user: Antoine Pitrou date: Sun Jan 18 17:39:32 2015 +0100 summary: Issue #23248: Update ssl error codes from latest OpenSSL git master. files: Misc/NEWS | 2 + Modules/_ssl_data.h | 297 ++++++++++++++++++++++++- Tools/ssl/make_ssl_data.py | 57 +++- 3 files changed, 336 insertions(+), 20 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Library ------- +- Issue #23248: Update ssl error codes from latest OpenSSL git master. + - Issue #23098: 64-bit dev_t is now supported in the os module. - Issue #23063: In the disutils' check command, fix parsing of reST with code or diff --git a/Modules/_ssl_data.h b/Modules/_ssl_data.h --- a/Modules/_ssl_data.h +++ b/Modules/_ssl_data.h @@ -1,5 +1,5 @@ /* File generated by Tools/ssl/make_ssl_data.py */ -/* Generated on 2012-05-16T23:56:40.981382 */ +/* Generated on 2015-01-17T20:33:43.377453 */ static struct py_ssl_library_code library_codes[] = { {"PEM", ERR_LIB_PEM}, @@ -179,6 +179,11 @@ #else {"BAD_CHECKSUM", ERR_LIB_SSL, 104}, #endif + #ifdef SSL_R_BAD_DATA + {"BAD_DATA", ERR_LIB_SSL, SSL_R_BAD_DATA}, + #else + {"BAD_DATA", ERR_LIB_SSL, 390}, + #endif #ifdef SSL_R_BAD_DATA_RETURNED_BY_CALLBACK {"BAD_DATA_RETURNED_BY_CALLBACK", ERR_LIB_SSL, SSL_R_BAD_DATA_RETURNED_BY_CALLBACK}, #else @@ -309,6 +314,46 @@ #else {"BAD_SIGNATURE", ERR_LIB_SSL, 123}, #endif + #ifdef SSL_R_BAD_SRP_A_LENGTH + {"BAD_SRP_A_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_A_LENGTH}, + #else + {"BAD_SRP_A_LENGTH", ERR_LIB_SSL, 347}, + #endif + #ifdef SSL_R_BAD_SRP_B_LENGTH + {"BAD_SRP_B_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_B_LENGTH}, + #else + {"BAD_SRP_B_LENGTH", ERR_LIB_SSL, 348}, + #endif + #ifdef SSL_R_BAD_SRP_G_LENGTH + {"BAD_SRP_G_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_G_LENGTH}, + #else + {"BAD_SRP_G_LENGTH", ERR_LIB_SSL, 349}, + #endif + #ifdef SSL_R_BAD_SRP_N_LENGTH + {"BAD_SRP_N_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_N_LENGTH}, + #else + {"BAD_SRP_N_LENGTH", ERR_LIB_SSL, 350}, + #endif + #ifdef SSL_R_BAD_SRP_PARAMETERS + {"BAD_SRP_PARAMETERS", ERR_LIB_SSL, SSL_R_BAD_SRP_PARAMETERS}, + #else + {"BAD_SRP_PARAMETERS", ERR_LIB_SSL, 371}, + #endif + #ifdef SSL_R_BAD_SRP_S_LENGTH + {"BAD_SRP_S_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_S_LENGTH}, + #else + {"BAD_SRP_S_LENGTH", ERR_LIB_SSL, 351}, + #endif + #ifdef SSL_R_BAD_SRTP_MKI_VALUE + {"BAD_SRTP_MKI_VALUE", ERR_LIB_SSL, SSL_R_BAD_SRTP_MKI_VALUE}, + #else + {"BAD_SRTP_MKI_VALUE", ERR_LIB_SSL, 352}, + #endif + #ifdef SSL_R_BAD_SRTP_PROTECTION_PROFILE_LIST + {"BAD_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, SSL_R_BAD_SRTP_PROTECTION_PROFILE_LIST}, + #else + {"BAD_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, 353}, + #endif #ifdef SSL_R_BAD_SSL_FILETYPE {"BAD_SSL_FILETYPE", ERR_LIB_SSL, SSL_R_BAD_SSL_FILETYPE}, #else @@ -324,6 +369,11 @@ #else {"BAD_STATE", ERR_LIB_SSL, 126}, #endif + #ifdef SSL_R_BAD_VALUE + {"BAD_VALUE", ERR_LIB_SSL, SSL_R_BAD_VALUE}, + #else + {"BAD_VALUE", ERR_LIB_SSL, 384}, + #endif #ifdef SSL_R_BAD_WRITE_RETRY {"BAD_WRITE_RETRY", ERR_LIB_SSL, SSL_R_BAD_WRITE_RETRY}, #else @@ -354,6 +404,16 @@ #else {"CA_DN_TOO_LONG", ERR_LIB_SSL, 132}, #endif + #ifdef SSL_R_CA_KEY_TOO_SMALL + {"CA_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_CA_KEY_TOO_SMALL}, + #else + {"CA_KEY_TOO_SMALL", ERR_LIB_SSL, 397}, + #endif + #ifdef SSL_R_CA_MD_TOO_WEAK + {"CA_MD_TOO_WEAK", ERR_LIB_SSL, SSL_R_CA_MD_TOO_WEAK}, + #else + {"CA_MD_TOO_WEAK", ERR_LIB_SSL, 398}, + #endif #ifdef SSL_R_CCS_RECEIVED_EARLY {"CCS_RECEIVED_EARLY", ERR_LIB_SSL, SSL_R_CCS_RECEIVED_EARLY}, #else @@ -364,6 +424,11 @@ #else {"CERTIFICATE_VERIFY_FAILED", ERR_LIB_SSL, 134}, #endif + #ifdef SSL_R_CERT_CB_ERROR + {"CERT_CB_ERROR", ERR_LIB_SSL, SSL_R_CERT_CB_ERROR}, + #else + {"CERT_CB_ERROR", ERR_LIB_SSL, 377}, + #endif #ifdef SSL_R_CERT_LENGTH_MISMATCH {"CERT_LENGTH_MISMATCH", ERR_LIB_SSL, SSL_R_CERT_LENGTH_MISMATCH}, #else @@ -454,6 +519,11 @@ #else {"DECRYPTION_FAILED_OR_BAD_RECORD_MAC", ERR_LIB_SSL, 281}, #endif + #ifdef SSL_R_DH_KEY_TOO_SMALL + {"DH_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_DH_KEY_TOO_SMALL}, + #else + {"DH_KEY_TOO_SMALL", ERR_LIB_SSL, 394}, + #endif #ifdef SSL_R_DH_PUBLIC_VALUE_LENGTH_IS_WRONG {"DH_PUBLIC_VALUE_LENGTH_IS_WRONG", ERR_LIB_SSL, SSL_R_DH_PUBLIC_VALUE_LENGTH_IS_WRONG}, #else @@ -494,11 +564,26 @@ #else {"ECC_CERT_SHOULD_HAVE_SHA1_SIGNATURE", ERR_LIB_SSL, 323}, #endif + #ifdef SSL_R_ECDH_REQUIRED_FOR_SUITEB_MODE + {"ECDH_REQUIRED_FOR_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ECDH_REQUIRED_FOR_SUITEB_MODE}, + #else + {"ECDH_REQUIRED_FOR_SUITEB_MODE", ERR_LIB_SSL, 374}, + #endif #ifdef SSL_R_ECGROUP_TOO_LARGE_FOR_CIPHER {"ECGROUP_TOO_LARGE_FOR_CIPHER", ERR_LIB_SSL, SSL_R_ECGROUP_TOO_LARGE_FOR_CIPHER}, #else {"ECGROUP_TOO_LARGE_FOR_CIPHER", ERR_LIB_SSL, 310}, #endif + #ifdef SSL_R_EE_KEY_TOO_SMALL + {"EE_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_EE_KEY_TOO_SMALL}, + #else + {"EE_KEY_TOO_SMALL", ERR_LIB_SSL, 399}, + #endif + #ifdef SSL_R_EMPTY_SRTP_PROTECTION_PROFILE_LIST + {"EMPTY_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, SSL_R_EMPTY_SRTP_PROTECTION_PROFILE_LIST}, + #else + {"EMPTY_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, 354}, + #endif #ifdef SSL_R_ENCRYPTED_LENGTH_TOO_LONG {"ENCRYPTED_LENGTH_TOO_LONG", ERR_LIB_SSL, SSL_R_ENCRYPTED_LENGTH_TOO_LONG}, #else @@ -529,6 +614,16 @@ #else {"GOT_A_FIN_BEFORE_A_CCS", ERR_LIB_SSL, 154}, #endif + #ifdef SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS + {"GOT_NEXT_PROTO_BEFORE_A_CCS", ERR_LIB_SSL, SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS}, + #else + {"GOT_NEXT_PROTO_BEFORE_A_CCS", ERR_LIB_SSL, 355}, + #endif + #ifdef SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION + {"GOT_NEXT_PROTO_WITHOUT_EXTENSION", ERR_LIB_SSL, SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION}, + #else + {"GOT_NEXT_PROTO_WITHOUT_EXTENSION", ERR_LIB_SSL, 356}, + #endif #ifdef SSL_R_HTTPS_PROXY_REQUEST {"HTTPS_PROXY_REQUEST", ERR_LIB_SSL, SSL_R_HTTPS_PROXY_REQUEST}, #else @@ -544,6 +639,16 @@ #else {"ILLEGAL_PADDING", ERR_LIB_SSL, 283}, #endif + #ifdef SSL_R_ILLEGAL_SUITEB_DIGEST + {"ILLEGAL_SUITEB_DIGEST", ERR_LIB_SSL, SSL_R_ILLEGAL_SUITEB_DIGEST}, + #else + {"ILLEGAL_SUITEB_DIGEST", ERR_LIB_SSL, 380}, + #endif + #ifdef SSL_R_INAPPROPRIATE_FALLBACK + {"INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, SSL_R_INAPPROPRIATE_FALLBACK}, + #else + {"INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, 373}, + #endif #ifdef SSL_R_INCONSISTENT_COMPRESSION {"INCONSISTENT_COMPRESSION", ERR_LIB_SSL, SSL_R_INCONSISTENT_COMPRESSION}, #else @@ -564,11 +669,26 @@ #else {"INVALID_COMPRESSION_ALGORITHM", ERR_LIB_SSL, 341}, #endif + #ifdef SSL_R_INVALID_NULL_CMD_NAME + {"INVALID_NULL_CMD_NAME", ERR_LIB_SSL, SSL_R_INVALID_NULL_CMD_NAME}, + #else + {"INVALID_NULL_CMD_NAME", ERR_LIB_SSL, 385}, + #endif #ifdef SSL_R_INVALID_PURPOSE {"INVALID_PURPOSE", ERR_LIB_SSL, SSL_R_INVALID_PURPOSE}, #else {"INVALID_PURPOSE", ERR_LIB_SSL, 278}, #endif + #ifdef SSL_R_INVALID_SERVERINFO_DATA + {"INVALID_SERVERINFO_DATA", ERR_LIB_SSL, SSL_R_INVALID_SERVERINFO_DATA}, + #else + {"INVALID_SERVERINFO_DATA", ERR_LIB_SSL, 388}, + #endif + #ifdef SSL_R_INVALID_SRP_USERNAME + {"INVALID_SRP_USERNAME", ERR_LIB_SSL, SSL_R_INVALID_SRP_USERNAME}, + #else + {"INVALID_SRP_USERNAME", ERR_LIB_SSL, 357}, + #endif #ifdef SSL_R_INVALID_STATUS_RESPONSE {"INVALID_STATUS_RESPONSE", ERR_LIB_SSL, SSL_R_INVALID_STATUS_RESPONSE}, #else @@ -689,6 +809,16 @@ #else {"MISSING_DSA_SIGNING_CERT", ERR_LIB_SSL, 165}, #endif + #ifdef SSL_R_MISSING_ECDH_CERT + {"MISSING_ECDH_CERT", ERR_LIB_SSL, SSL_R_MISSING_ECDH_CERT}, + #else + {"MISSING_ECDH_CERT", ERR_LIB_SSL, 382}, + #endif + #ifdef SSL_R_MISSING_ECDSA_SIGNING_CERT + {"MISSING_ECDSA_SIGNING_CERT", ERR_LIB_SSL, SSL_R_MISSING_ECDSA_SIGNING_CERT}, + #else + {"MISSING_ECDSA_SIGNING_CERT", ERR_LIB_SSL, 381}, + #endif #ifdef SSL_R_MISSING_EXPORT_TMP_DH_KEY {"MISSING_EXPORT_TMP_DH_KEY", ERR_LIB_SSL, SSL_R_MISSING_EXPORT_TMP_DH_KEY}, #else @@ -714,6 +844,11 @@ #else {"MISSING_RSA_SIGNING_CERT", ERR_LIB_SSL, 170}, #endif + #ifdef SSL_R_MISSING_SRP_PARAM + {"MISSING_SRP_PARAM", ERR_LIB_SSL, SSL_R_MISSING_SRP_PARAM}, + #else + {"MISSING_SRP_PARAM", ERR_LIB_SSL, 358}, + #endif #ifdef SSL_R_MISSING_TMP_DH_KEY {"MISSING_TMP_DH_KEY", ERR_LIB_SSL, SSL_R_MISSING_TMP_DH_KEY}, #else @@ -739,6 +874,11 @@ #else {"MISSING_VERIFY_MESSAGE", ERR_LIB_SSL, 174}, #endif + #ifdef SSL_R_MULTIPLE_SGC_RESTARTS + {"MULTIPLE_SGC_RESTARTS", ERR_LIB_SSL, SSL_R_MULTIPLE_SGC_RESTARTS}, + #else + {"MULTIPLE_SGC_RESTARTS", ERR_LIB_SSL, 346}, + #endif #ifdef SSL_R_NON_SSLV2_INITIAL_PACKET {"NON_SSLV2_INITIAL_PACKET", ERR_LIB_SSL, SSL_R_NON_SSLV2_INITIAL_PACKET}, #else @@ -819,6 +959,11 @@ #else {"NO_METHOD_SPECIFIED", ERR_LIB_SSL, 188}, #endif + #ifdef SSL_R_NO_PEM_EXTENSIONS + {"NO_PEM_EXTENSIONS", ERR_LIB_SSL, SSL_R_NO_PEM_EXTENSIONS}, + #else + {"NO_PEM_EXTENSIONS", ERR_LIB_SSL, 389}, + #endif #ifdef SSL_R_NO_PRIVATEKEY {"NO_PRIVATEKEY", ERR_LIB_SSL, SSL_R_NO_PRIVATEKEY}, #else @@ -854,6 +999,16 @@ #else {"NO_SHARED_CIPHER", ERR_LIB_SSL, 193}, #endif + #ifdef SSL_R_NO_SHARED_SIGATURE_ALGORITHMS + {"NO_SHARED_SIGATURE_ALGORITHMS", ERR_LIB_SSL, SSL_R_NO_SHARED_SIGATURE_ALGORITHMS}, + #else + {"NO_SHARED_SIGATURE_ALGORITHMS", ERR_LIB_SSL, 376}, + #endif + #ifdef SSL_R_NO_SRTP_PROFILES + {"NO_SRTP_PROFILES", ERR_LIB_SSL, SSL_R_NO_SRTP_PROFILES}, + #else + {"NO_SRTP_PROFILES", ERR_LIB_SSL, 359}, + #endif #ifdef SSL_R_NO_VERIFY_CALLBACK {"NO_VERIFY_CALLBACK", ERR_LIB_SSL, SSL_R_NO_VERIFY_CALLBACK}, #else @@ -879,6 +1034,16 @@ #else {"OLD_SESSION_COMPRESSION_ALGORITHM_NOT_RETURNED", ERR_LIB_SSL, 344}, #endif + #ifdef SSL_R_ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE + {"ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE}, + #else + {"ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, 387}, + #endif + #ifdef SSL_R_ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE + {"ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE}, + #else + {"ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, 379}, + #endif #ifdef SSL_R_ONLY_TLS_ALLOWED_IN_FIPS_MODE {"ONLY_TLS_ALLOWED_IN_FIPS_MODE", ERR_LIB_SSL, SSL_R_ONLY_TLS_ALLOWED_IN_FIPS_MODE}, #else @@ -934,6 +1099,16 @@ #else {"PEER_ERROR_UNSUPPORTED_CERTIFICATE_TYPE", ERR_LIB_SSL, 204}, #endif + #ifdef SSL_R_PEM_NAME_BAD_PREFIX + {"PEM_NAME_BAD_PREFIX", ERR_LIB_SSL, SSL_R_PEM_NAME_BAD_PREFIX}, + #else + {"PEM_NAME_BAD_PREFIX", ERR_LIB_SSL, 391}, + #endif + #ifdef SSL_R_PEM_NAME_TOO_SHORT + {"PEM_NAME_TOO_SHORT", ERR_LIB_SSL, SSL_R_PEM_NAME_TOO_SHORT}, + #else + {"PEM_NAME_TOO_SHORT", ERR_LIB_SSL, 392}, + #endif #ifdef SSL_R_PRE_MAC_LENGTH_TOO_LONG {"PRE_MAC_LENGTH_TOO_LONG", ERR_LIB_SSL, SSL_R_PRE_MAC_LENGTH_TOO_LONG}, #else @@ -1069,11 +1244,36 @@ #else {"SHORT_READ", ERR_LIB_SSL, 219}, #endif + #ifdef SSL_R_SIGNATURE_ALGORITHMS_ERROR + {"SIGNATURE_ALGORITHMS_ERROR", ERR_LIB_SSL, SSL_R_SIGNATURE_ALGORITHMS_ERROR}, + #else + {"SIGNATURE_ALGORITHMS_ERROR", ERR_LIB_SSL, 360}, + #endif #ifdef SSL_R_SIGNATURE_FOR_NON_SIGNING_CERTIFICATE {"SIGNATURE_FOR_NON_SIGNING_CERTIFICATE", ERR_LIB_SSL, SSL_R_SIGNATURE_FOR_NON_SIGNING_CERTIFICATE}, #else {"SIGNATURE_FOR_NON_SIGNING_CERTIFICATE", ERR_LIB_SSL, 220}, #endif + #ifdef SSL_R_SRP_A_CALC + {"SRP_A_CALC", ERR_LIB_SSL, SSL_R_SRP_A_CALC}, + #else + {"SRP_A_CALC", ERR_LIB_SSL, 361}, + #endif + #ifdef SSL_R_SRTP_COULD_NOT_ALLOCATE_PROFILES + {"SRTP_COULD_NOT_ALLOCATE_PROFILES", ERR_LIB_SSL, SSL_R_SRTP_COULD_NOT_ALLOCATE_PROFILES}, + #else + {"SRTP_COULD_NOT_ALLOCATE_PROFILES", ERR_LIB_SSL, 362}, + #endif + #ifdef SSL_R_SRTP_PROTECTION_PROFILE_LIST_TOO_LONG + {"SRTP_PROTECTION_PROFILE_LIST_TOO_LONG", ERR_LIB_SSL, SSL_R_SRTP_PROTECTION_PROFILE_LIST_TOO_LONG}, + #else + {"SRTP_PROTECTION_PROFILE_LIST_TOO_LONG", ERR_LIB_SSL, 363}, + #endif + #ifdef SSL_R_SRTP_UNKNOWN_PROTECTION_PROFILE + {"SRTP_UNKNOWN_PROTECTION_PROFILE", ERR_LIB_SSL, SSL_R_SRTP_UNKNOWN_PROTECTION_PROFILE}, + #else + {"SRTP_UNKNOWN_PROTECTION_PROFILE", ERR_LIB_SSL, 364}, + #endif #ifdef SSL_R_SSL23_DOING_SESSION_ID_REUSE {"SSL23_DOING_SESSION_ID_REUSE", ERR_LIB_SSL, SSL_R_SSL23_DOING_SESSION_ID_REUSE}, #else @@ -1179,6 +1379,11 @@ #else {"SSL_LIBRARY_HAS_NO_CIPHERS", ERR_LIB_SSL, 230}, #endif + #ifdef SSL_R_SSL_NEGATIVE_LENGTH + {"SSL_NEGATIVE_LENGTH", ERR_LIB_SSL, SSL_R_SSL_NEGATIVE_LENGTH}, + #else + {"SSL_NEGATIVE_LENGTH", ERR_LIB_SSL, 372}, + #endif #ifdef SSL_R_SSL_SESSION_ID_CALLBACK_FAILED {"SSL_SESSION_ID_CALLBACK_FAILED", ERR_LIB_SSL, SSL_R_SSL_SESSION_ID_CALLBACK_FAILED}, #else @@ -1229,6 +1434,11 @@ #else {"TLSV1_ALERT_EXPORT_RESTRICTION", ERR_LIB_SSL, 1060}, #endif + #ifdef SSL_R_TLSV1_ALERT_INAPPROPRIATE_FALLBACK + {"TLSV1_ALERT_INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, SSL_R_TLSV1_ALERT_INAPPROPRIATE_FALLBACK}, + #else + {"TLSV1_ALERT_INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, 1086}, + #endif #ifdef SSL_R_TLSV1_ALERT_INSUFFICIENT_SECURITY {"TLSV1_ALERT_INSUFFICIENT_SECURITY", ERR_LIB_SSL, SSL_R_TLSV1_ALERT_INSUFFICIENT_SECURITY}, #else @@ -1294,6 +1504,21 @@ #else {"TLS_CLIENT_CERT_REQ_WITH_ANON_CIPHER", ERR_LIB_SSL, 232}, #endif + #ifdef SSL_R_TLS_HEARTBEAT_PEER_DOESNT_ACCEPT + {"TLS_HEARTBEAT_PEER_DOESNT_ACCEPT", ERR_LIB_SSL, SSL_R_TLS_HEARTBEAT_PEER_DOESNT_ACCEPT}, + #else + {"TLS_HEARTBEAT_PEER_DOESNT_ACCEPT", ERR_LIB_SSL, 365}, + #endif + #ifdef SSL_R_TLS_HEARTBEAT_PENDING + {"TLS_HEARTBEAT_PENDING", ERR_LIB_SSL, SSL_R_TLS_HEARTBEAT_PENDING}, + #else + {"TLS_HEARTBEAT_PENDING", ERR_LIB_SSL, 366}, + #endif + #ifdef SSL_R_TLS_ILLEGAL_EXPORTER_LABEL + {"TLS_ILLEGAL_EXPORTER_LABEL", ERR_LIB_SSL, SSL_R_TLS_ILLEGAL_EXPORTER_LABEL}, + #else + {"TLS_ILLEGAL_EXPORTER_LABEL", ERR_LIB_SSL, 367}, + #endif #ifdef SSL_R_TLS_INVALID_ECPOINTFORMAT_LIST {"TLS_INVALID_ECPOINTFORMAT_LIST", ERR_LIB_SSL, SSL_R_TLS_INVALID_ECPOINTFORMAT_LIST}, #else @@ -1399,6 +1624,16 @@ #else {"UNKNOWN_CIPHER_TYPE", ERR_LIB_SSL, 249}, #endif + #ifdef SSL_R_UNKNOWN_CMD_NAME + {"UNKNOWN_CMD_NAME", ERR_LIB_SSL, SSL_R_UNKNOWN_CMD_NAME}, + #else + {"UNKNOWN_CMD_NAME", ERR_LIB_SSL, 386}, + #endif + #ifdef SSL_R_UNKNOWN_DIGEST + {"UNKNOWN_DIGEST", ERR_LIB_SSL, SSL_R_UNKNOWN_DIGEST}, + #else + {"UNKNOWN_DIGEST", ERR_LIB_SSL, 368}, + #endif #ifdef SSL_R_UNKNOWN_KEY_EXCHANGE_TYPE {"UNKNOWN_KEY_EXCHANGE_TYPE", ERR_LIB_SSL, SSL_R_UNKNOWN_KEY_EXCHANGE_TYPE}, #else @@ -1469,16 +1704,36 @@ #else {"UNSUPPORTED_STATUS_TYPE", ERR_LIB_SSL, 329}, #endif + #ifdef SSL_R_USE_SRTP_NOT_NEGOTIATED + {"USE_SRTP_NOT_NEGOTIATED", ERR_LIB_SSL, SSL_R_USE_SRTP_NOT_NEGOTIATED}, + #else + {"USE_SRTP_NOT_NEGOTIATED", ERR_LIB_SSL, 369}, + #endif + #ifdef SSL_R_VERSION_TOO_LOW + {"VERSION_TOO_LOW", ERR_LIB_SSL, SSL_R_VERSION_TOO_LOW}, + #else + {"VERSION_TOO_LOW", ERR_LIB_SSL, 396}, + #endif #ifdef SSL_R_WRITE_BIO_NOT_SET {"WRITE_BIO_NOT_SET", ERR_LIB_SSL, SSL_R_WRITE_BIO_NOT_SET}, #else {"WRITE_BIO_NOT_SET", ERR_LIB_SSL, 260}, #endif + #ifdef SSL_R_WRONG_CERTIFICATE_TYPE + {"WRONG_CERTIFICATE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_CERTIFICATE_TYPE}, + #else + {"WRONG_CERTIFICATE_TYPE", ERR_LIB_SSL, 383}, + #endif #ifdef SSL_R_WRONG_CIPHER_RETURNED {"WRONG_CIPHER_RETURNED", ERR_LIB_SSL, SSL_R_WRONG_CIPHER_RETURNED}, #else {"WRONG_CIPHER_RETURNED", ERR_LIB_SSL, 261}, #endif + #ifdef SSL_R_WRONG_CURVE + {"WRONG_CURVE", ERR_LIB_SSL, SSL_R_WRONG_CURVE}, + #else + {"WRONG_CURVE", ERR_LIB_SSL, 378}, + #endif #ifdef SSL_R_WRONG_MESSAGE_TYPE {"WRONG_MESSAGE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_MESSAGE_TYPE}, #else @@ -1499,6 +1754,11 @@ #else {"WRONG_SIGNATURE_SIZE", ERR_LIB_SSL, 265}, #endif + #ifdef SSL_R_WRONG_SIGNATURE_TYPE + {"WRONG_SIGNATURE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_SIGNATURE_TYPE}, + #else + {"WRONG_SIGNATURE_TYPE", ERR_LIB_SSL, 370}, + #endif #ifdef SSL_R_WRONG_SSL_VERSION {"WRONG_SSL_VERSION", ERR_LIB_SSL, SSL_R_WRONG_SSL_VERSION}, #else @@ -1519,6 +1779,11 @@ #else {"X509_VERIFICATION_SETUP_PROBLEMS", ERR_LIB_SSL, 269}, #endif + #ifdef X509_R_AKID_MISMATCH + {"AKID_MISMATCH", ERR_LIB_X509, X509_R_AKID_MISMATCH}, + #else + {"AKID_MISMATCH", ERR_LIB_X509, 110}, + #endif #ifdef X509_R_BAD_X509_FILETYPE {"BAD_X509_FILETYPE", ERR_LIB_X509, X509_R_BAD_X509_FILETYPE}, #else @@ -1539,11 +1804,26 @@ #else {"CERT_ALREADY_IN_HASH_TABLE", ERR_LIB_X509, 101}, #endif + #ifdef X509_R_CRL_ALREADY_DELTA + {"CRL_ALREADY_DELTA", ERR_LIB_X509, X509_R_CRL_ALREADY_DELTA}, + #else + {"CRL_ALREADY_DELTA", ERR_LIB_X509, 127}, + #endif + #ifdef X509_R_CRL_VERIFY_FAILURE + {"CRL_VERIFY_FAILURE", ERR_LIB_X509, X509_R_CRL_VERIFY_FAILURE}, + #else + {"CRL_VERIFY_FAILURE", ERR_LIB_X509, 131}, + #endif #ifdef X509_R_ERR_ASN1_LIB {"ERR_ASN1_LIB", ERR_LIB_X509, X509_R_ERR_ASN1_LIB}, #else {"ERR_ASN1_LIB", ERR_LIB_X509, 102}, #endif + #ifdef X509_R_IDP_MISMATCH + {"IDP_MISMATCH", ERR_LIB_X509, X509_R_IDP_MISMATCH}, + #else + {"IDP_MISMATCH", ERR_LIB_X509, 128}, + #endif #ifdef X509_R_INVALID_DIRECTORY {"INVALID_DIRECTORY", ERR_LIB_X509, X509_R_INVALID_DIRECTORY}, #else @@ -1559,6 +1839,11 @@ #else {"INVALID_TRUST", ERR_LIB_X509, 123}, #endif + #ifdef X509_R_ISSUER_MISMATCH + {"ISSUER_MISMATCH", ERR_LIB_X509, X509_R_ISSUER_MISMATCH}, + #else + {"ISSUER_MISMATCH", ERR_LIB_X509, 129}, + #endif #ifdef X509_R_KEY_TYPE_MISMATCH {"KEY_TYPE_MISMATCH", ERR_LIB_X509, X509_R_KEY_TYPE_MISMATCH}, #else @@ -1584,11 +1869,21 @@ #else {"METHOD_NOT_SUPPORTED", ERR_LIB_X509, 124}, #endif + #ifdef X509_R_NEWER_CRL_NOT_NEWER + {"NEWER_CRL_NOT_NEWER", ERR_LIB_X509, X509_R_NEWER_CRL_NOT_NEWER}, + #else + {"NEWER_CRL_NOT_NEWER", ERR_LIB_X509, 132}, + #endif #ifdef X509_R_NO_CERT_SET_FOR_US_TO_VERIFY {"NO_CERT_SET_FOR_US_TO_VERIFY", ERR_LIB_X509, X509_R_NO_CERT_SET_FOR_US_TO_VERIFY}, #else {"NO_CERT_SET_FOR_US_TO_VERIFY", ERR_LIB_X509, 105}, #endif + #ifdef X509_R_NO_CRL_NUMBER + {"NO_CRL_NUMBER", ERR_LIB_X509, X509_R_NO_CRL_NUMBER}, + #else + {"NO_CRL_NUMBER", ERR_LIB_X509, 130}, + #endif #ifdef X509_R_PUBLIC_KEY_DECODE_ERROR {"PUBLIC_KEY_DECODE_ERROR", ERR_LIB_X509, X509_R_PUBLIC_KEY_DECODE_ERROR}, #else diff --git a/Tools/ssl/make_ssl_data.py b/Tools/ssl/make_ssl_data.py --- a/Tools/ssl/make_ssl_data.py +++ b/Tools/ssl/make_ssl_data.py @@ -5,8 +5,7 @@ `library` and `reason` mnemnonics to a more recent OpenSSL version. It takes two arguments: -- the path to the OpenSSL include files' directory - (e.g. openssl-1.0.1-beta3/include/openssl/) +- the path to the OpenSSL source tree (e.g. git checkout) - the path to the C file to be generated (probably Modules/_ssl_data.h) """ @@ -15,9 +14,10 @@ import os import re import sys +import _ssl -def parse_error_codes(h_file, prefix): +def parse_error_codes(h_file, prefix, libcode): pat = re.compile(r"#define\W+(%s([\w]+))\W+(\d+)\b" % re.escape(prefix)) codes = [] with open(h_file, "r", encoding="latin1") as f: @@ -26,7 +26,8 @@ if match: code, name, num = match.groups() num = int(num) - codes.append((code, name, num)) + # e.g. ("SSL_R_BAD_DATA", ("ERR_LIB_SSL", "BAD_DATA", 390)) + codes.append((code, (libcode, name, num))) return codes if __name__ == "__main__": @@ -34,12 +35,32 @@ outfile = sys.argv[2] use_stdout = outfile == '-' f = sys.stdout if use_stdout else open(outfile, "w") - error_libraries = ( - # (library code, mnemonic, error prefix, header file) - ('ERR_LIB_PEM', 'PEM', 'PEM_R_', 'pem.h'), - ('ERR_LIB_SSL', 'SSL', 'SSL_R_', 'ssl.h'), - ('ERR_LIB_X509', 'X509', 'X509_R_', 'x509.h'), - ) + error_libraries = { + # mnemonic -> (library code, error prefix, header file) + 'PEM': ('ERR_LIB_PEM', 'PEM_R_', 'crypto/pem/pem.h'), + 'SSL': ('ERR_LIB_SSL', 'SSL_R_', 'ssl/ssl.h'), + 'X509': ('ERR_LIB_X509', 'X509_R_', 'crypto/x509/x509.h'), + } + + # Read codes from libraries + new_codes = [] + for libcode, prefix, h_file in sorted(error_libraries.values()): + new_codes += parse_error_codes(os.path.join(openssl_inc, h_file), + prefix, libcode) + new_code_nums = set((libcode, num) + for (code, (libcode, name, num)) in new_codes) + + # Merge with existing codes (in case some old codes disappeared). + codes = {} + for errname, (libnum, errnum) in _ssl.err_names_to_codes.items(): + lib = error_libraries[_ssl.lib_codes_to_names[libnum]] + libcode = lib[0] # e.g. ERR_LIB_PEM + errcode = lib[1] + errname # e.g. SSL_R_BAD_SSL_SESSION_ID_LENGTH + # Only keep it if the numeric codes weren't reused + if (libcode, errnum) not in new_code_nums: + codes[errcode] = libcode, errname, errnum + codes.update(dict(new_codes)) + def w(l): f.write(l + "\n") w("/* File generated by Tools/ssl/make_ssl_data.py */") @@ -47,21 +68,19 @@ w("") w("static struct py_ssl_library_code library_codes[] = {") - for libcode, mnemo, _, _ in error_libraries: + for mnemo, (libcode, _, _) in sorted(error_libraries.items()): w(' {"%s", %s},' % (mnemo, libcode)) w(' { NULL }') w('};') w("") w("static struct py_ssl_error_code error_codes[] = {") - for libcode, _, prefix, h_file in error_libraries: - codes = parse_error_codes(os.path.join(openssl_inc, h_file), prefix) - for code, name, num in sorted(codes): - w(' #ifdef %s' % (code)) - w(' {"%s", %s, %s},' % (name, libcode, code)) - w(' #else') - w(' {"%s", %s, %d},' % (name, libcode, num)) - w(' #endif') + for errcode, (libcode, name, num) in sorted(codes.items()): + w(' #ifdef %s' % (errcode)) + w(' {"%s", %s, %s},' % (name, libcode, errcode)) + w(' #else') + w(' {"%s", %s, %d},' % (name, libcode, num)) + w(' #endif') w(' { NULL }') w('};') if not use_stdout: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 17:42:32 2015 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 18 Jan 2015 16:42:32 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjQ4?= =?utf-8?q?=3A_Update_ssl_error_codes_from_latest_OpenSSL_git_master=2E?= Message-ID: <20150118164231.118100.50486@psf.io> https://hg.python.org/cpython/rev/2eea364c2863 changeset: 94209:2eea364c2863 branch: 3.4 parent: 94205:0353c7e5e0c2 user: Antoine Pitrou date: Sun Jan 18 17:39:32 2015 +0100 summary: Issue #23248: Update ssl error codes from latest OpenSSL git master. files: Misc/NEWS | 2 + Modules/_ssl_data.h | 297 ++++++++++++++++++++++++- Tools/ssl/make_ssl_data.py | 57 +++- 3 files changed, 336 insertions(+), 20 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #23248: Update ssl error codes from latest OpenSSL git master. + - Issue #23098: 64-bit dev_t is now supported in the os module. - Issue #23250: In the http.cookies module, capitalize "HttpOnly" and "Secure" diff --git a/Modules/_ssl_data.h b/Modules/_ssl_data.h --- a/Modules/_ssl_data.h +++ b/Modules/_ssl_data.h @@ -1,5 +1,5 @@ /* File generated by Tools/ssl/make_ssl_data.py */ -/* Generated on 2012-05-16T23:56:40.981382 */ +/* Generated on 2015-01-17T20:33:43.377453 */ static struct py_ssl_library_code library_codes[] = { {"PEM", ERR_LIB_PEM}, @@ -179,6 +179,11 @@ #else {"BAD_CHECKSUM", ERR_LIB_SSL, 104}, #endif + #ifdef SSL_R_BAD_DATA + {"BAD_DATA", ERR_LIB_SSL, SSL_R_BAD_DATA}, + #else + {"BAD_DATA", ERR_LIB_SSL, 390}, + #endif #ifdef SSL_R_BAD_DATA_RETURNED_BY_CALLBACK {"BAD_DATA_RETURNED_BY_CALLBACK", ERR_LIB_SSL, SSL_R_BAD_DATA_RETURNED_BY_CALLBACK}, #else @@ -309,6 +314,46 @@ #else {"BAD_SIGNATURE", ERR_LIB_SSL, 123}, #endif + #ifdef SSL_R_BAD_SRP_A_LENGTH + {"BAD_SRP_A_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_A_LENGTH}, + #else + {"BAD_SRP_A_LENGTH", ERR_LIB_SSL, 347}, + #endif + #ifdef SSL_R_BAD_SRP_B_LENGTH + {"BAD_SRP_B_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_B_LENGTH}, + #else + {"BAD_SRP_B_LENGTH", ERR_LIB_SSL, 348}, + #endif + #ifdef SSL_R_BAD_SRP_G_LENGTH + {"BAD_SRP_G_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_G_LENGTH}, + #else + {"BAD_SRP_G_LENGTH", ERR_LIB_SSL, 349}, + #endif + #ifdef SSL_R_BAD_SRP_N_LENGTH + {"BAD_SRP_N_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_N_LENGTH}, + #else + {"BAD_SRP_N_LENGTH", ERR_LIB_SSL, 350}, + #endif + #ifdef SSL_R_BAD_SRP_PARAMETERS + {"BAD_SRP_PARAMETERS", ERR_LIB_SSL, SSL_R_BAD_SRP_PARAMETERS}, + #else + {"BAD_SRP_PARAMETERS", ERR_LIB_SSL, 371}, + #endif + #ifdef SSL_R_BAD_SRP_S_LENGTH + {"BAD_SRP_S_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_S_LENGTH}, + #else + {"BAD_SRP_S_LENGTH", ERR_LIB_SSL, 351}, + #endif + #ifdef SSL_R_BAD_SRTP_MKI_VALUE + {"BAD_SRTP_MKI_VALUE", ERR_LIB_SSL, SSL_R_BAD_SRTP_MKI_VALUE}, + #else + {"BAD_SRTP_MKI_VALUE", ERR_LIB_SSL, 352}, + #endif + #ifdef SSL_R_BAD_SRTP_PROTECTION_PROFILE_LIST + {"BAD_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, SSL_R_BAD_SRTP_PROTECTION_PROFILE_LIST}, + #else + {"BAD_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, 353}, + #endif #ifdef SSL_R_BAD_SSL_FILETYPE {"BAD_SSL_FILETYPE", ERR_LIB_SSL, SSL_R_BAD_SSL_FILETYPE}, #else @@ -324,6 +369,11 @@ #else {"BAD_STATE", ERR_LIB_SSL, 126}, #endif + #ifdef SSL_R_BAD_VALUE + {"BAD_VALUE", ERR_LIB_SSL, SSL_R_BAD_VALUE}, + #else + {"BAD_VALUE", ERR_LIB_SSL, 384}, + #endif #ifdef SSL_R_BAD_WRITE_RETRY {"BAD_WRITE_RETRY", ERR_LIB_SSL, SSL_R_BAD_WRITE_RETRY}, #else @@ -354,6 +404,16 @@ #else {"CA_DN_TOO_LONG", ERR_LIB_SSL, 132}, #endif + #ifdef SSL_R_CA_KEY_TOO_SMALL + {"CA_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_CA_KEY_TOO_SMALL}, + #else + {"CA_KEY_TOO_SMALL", ERR_LIB_SSL, 397}, + #endif + #ifdef SSL_R_CA_MD_TOO_WEAK + {"CA_MD_TOO_WEAK", ERR_LIB_SSL, SSL_R_CA_MD_TOO_WEAK}, + #else + {"CA_MD_TOO_WEAK", ERR_LIB_SSL, 398}, + #endif #ifdef SSL_R_CCS_RECEIVED_EARLY {"CCS_RECEIVED_EARLY", ERR_LIB_SSL, SSL_R_CCS_RECEIVED_EARLY}, #else @@ -364,6 +424,11 @@ #else {"CERTIFICATE_VERIFY_FAILED", ERR_LIB_SSL, 134}, #endif + #ifdef SSL_R_CERT_CB_ERROR + {"CERT_CB_ERROR", ERR_LIB_SSL, SSL_R_CERT_CB_ERROR}, + #else + {"CERT_CB_ERROR", ERR_LIB_SSL, 377}, + #endif #ifdef SSL_R_CERT_LENGTH_MISMATCH {"CERT_LENGTH_MISMATCH", ERR_LIB_SSL, SSL_R_CERT_LENGTH_MISMATCH}, #else @@ -454,6 +519,11 @@ #else {"DECRYPTION_FAILED_OR_BAD_RECORD_MAC", ERR_LIB_SSL, 281}, #endif + #ifdef SSL_R_DH_KEY_TOO_SMALL + {"DH_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_DH_KEY_TOO_SMALL}, + #else + {"DH_KEY_TOO_SMALL", ERR_LIB_SSL, 394}, + #endif #ifdef SSL_R_DH_PUBLIC_VALUE_LENGTH_IS_WRONG {"DH_PUBLIC_VALUE_LENGTH_IS_WRONG", ERR_LIB_SSL, SSL_R_DH_PUBLIC_VALUE_LENGTH_IS_WRONG}, #else @@ -494,11 +564,26 @@ #else {"ECC_CERT_SHOULD_HAVE_SHA1_SIGNATURE", ERR_LIB_SSL, 323}, #endif + #ifdef SSL_R_ECDH_REQUIRED_FOR_SUITEB_MODE + {"ECDH_REQUIRED_FOR_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ECDH_REQUIRED_FOR_SUITEB_MODE}, + #else + {"ECDH_REQUIRED_FOR_SUITEB_MODE", ERR_LIB_SSL, 374}, + #endif #ifdef SSL_R_ECGROUP_TOO_LARGE_FOR_CIPHER {"ECGROUP_TOO_LARGE_FOR_CIPHER", ERR_LIB_SSL, SSL_R_ECGROUP_TOO_LARGE_FOR_CIPHER}, #else {"ECGROUP_TOO_LARGE_FOR_CIPHER", ERR_LIB_SSL, 310}, #endif + #ifdef SSL_R_EE_KEY_TOO_SMALL + {"EE_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_EE_KEY_TOO_SMALL}, + #else + {"EE_KEY_TOO_SMALL", ERR_LIB_SSL, 399}, + #endif + #ifdef SSL_R_EMPTY_SRTP_PROTECTION_PROFILE_LIST + {"EMPTY_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, SSL_R_EMPTY_SRTP_PROTECTION_PROFILE_LIST}, + #else + {"EMPTY_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, 354}, + #endif #ifdef SSL_R_ENCRYPTED_LENGTH_TOO_LONG {"ENCRYPTED_LENGTH_TOO_LONG", ERR_LIB_SSL, SSL_R_ENCRYPTED_LENGTH_TOO_LONG}, #else @@ -529,6 +614,16 @@ #else {"GOT_A_FIN_BEFORE_A_CCS", ERR_LIB_SSL, 154}, #endif + #ifdef SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS + {"GOT_NEXT_PROTO_BEFORE_A_CCS", ERR_LIB_SSL, SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS}, + #else + {"GOT_NEXT_PROTO_BEFORE_A_CCS", ERR_LIB_SSL, 355}, + #endif + #ifdef SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION + {"GOT_NEXT_PROTO_WITHOUT_EXTENSION", ERR_LIB_SSL, SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION}, + #else + {"GOT_NEXT_PROTO_WITHOUT_EXTENSION", ERR_LIB_SSL, 356}, + #endif #ifdef SSL_R_HTTPS_PROXY_REQUEST {"HTTPS_PROXY_REQUEST", ERR_LIB_SSL, SSL_R_HTTPS_PROXY_REQUEST}, #else @@ -544,6 +639,16 @@ #else {"ILLEGAL_PADDING", ERR_LIB_SSL, 283}, #endif + #ifdef SSL_R_ILLEGAL_SUITEB_DIGEST + {"ILLEGAL_SUITEB_DIGEST", ERR_LIB_SSL, SSL_R_ILLEGAL_SUITEB_DIGEST}, + #else + {"ILLEGAL_SUITEB_DIGEST", ERR_LIB_SSL, 380}, + #endif + #ifdef SSL_R_INAPPROPRIATE_FALLBACK + {"INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, SSL_R_INAPPROPRIATE_FALLBACK}, + #else + {"INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, 373}, + #endif #ifdef SSL_R_INCONSISTENT_COMPRESSION {"INCONSISTENT_COMPRESSION", ERR_LIB_SSL, SSL_R_INCONSISTENT_COMPRESSION}, #else @@ -564,11 +669,26 @@ #else {"INVALID_COMPRESSION_ALGORITHM", ERR_LIB_SSL, 341}, #endif + #ifdef SSL_R_INVALID_NULL_CMD_NAME + {"INVALID_NULL_CMD_NAME", ERR_LIB_SSL, SSL_R_INVALID_NULL_CMD_NAME}, + #else + {"INVALID_NULL_CMD_NAME", ERR_LIB_SSL, 385}, + #endif #ifdef SSL_R_INVALID_PURPOSE {"INVALID_PURPOSE", ERR_LIB_SSL, SSL_R_INVALID_PURPOSE}, #else {"INVALID_PURPOSE", ERR_LIB_SSL, 278}, #endif + #ifdef SSL_R_INVALID_SERVERINFO_DATA + {"INVALID_SERVERINFO_DATA", ERR_LIB_SSL, SSL_R_INVALID_SERVERINFO_DATA}, + #else + {"INVALID_SERVERINFO_DATA", ERR_LIB_SSL, 388}, + #endif + #ifdef SSL_R_INVALID_SRP_USERNAME + {"INVALID_SRP_USERNAME", ERR_LIB_SSL, SSL_R_INVALID_SRP_USERNAME}, + #else + {"INVALID_SRP_USERNAME", ERR_LIB_SSL, 357}, + #endif #ifdef SSL_R_INVALID_STATUS_RESPONSE {"INVALID_STATUS_RESPONSE", ERR_LIB_SSL, SSL_R_INVALID_STATUS_RESPONSE}, #else @@ -689,6 +809,16 @@ #else {"MISSING_DSA_SIGNING_CERT", ERR_LIB_SSL, 165}, #endif + #ifdef SSL_R_MISSING_ECDH_CERT + {"MISSING_ECDH_CERT", ERR_LIB_SSL, SSL_R_MISSING_ECDH_CERT}, + #else + {"MISSING_ECDH_CERT", ERR_LIB_SSL, 382}, + #endif + #ifdef SSL_R_MISSING_ECDSA_SIGNING_CERT + {"MISSING_ECDSA_SIGNING_CERT", ERR_LIB_SSL, SSL_R_MISSING_ECDSA_SIGNING_CERT}, + #else + {"MISSING_ECDSA_SIGNING_CERT", ERR_LIB_SSL, 381}, + #endif #ifdef SSL_R_MISSING_EXPORT_TMP_DH_KEY {"MISSING_EXPORT_TMP_DH_KEY", ERR_LIB_SSL, SSL_R_MISSING_EXPORT_TMP_DH_KEY}, #else @@ -714,6 +844,11 @@ #else {"MISSING_RSA_SIGNING_CERT", ERR_LIB_SSL, 170}, #endif + #ifdef SSL_R_MISSING_SRP_PARAM + {"MISSING_SRP_PARAM", ERR_LIB_SSL, SSL_R_MISSING_SRP_PARAM}, + #else + {"MISSING_SRP_PARAM", ERR_LIB_SSL, 358}, + #endif #ifdef SSL_R_MISSING_TMP_DH_KEY {"MISSING_TMP_DH_KEY", ERR_LIB_SSL, SSL_R_MISSING_TMP_DH_KEY}, #else @@ -739,6 +874,11 @@ #else {"MISSING_VERIFY_MESSAGE", ERR_LIB_SSL, 174}, #endif + #ifdef SSL_R_MULTIPLE_SGC_RESTARTS + {"MULTIPLE_SGC_RESTARTS", ERR_LIB_SSL, SSL_R_MULTIPLE_SGC_RESTARTS}, + #else + {"MULTIPLE_SGC_RESTARTS", ERR_LIB_SSL, 346}, + #endif #ifdef SSL_R_NON_SSLV2_INITIAL_PACKET {"NON_SSLV2_INITIAL_PACKET", ERR_LIB_SSL, SSL_R_NON_SSLV2_INITIAL_PACKET}, #else @@ -819,6 +959,11 @@ #else {"NO_METHOD_SPECIFIED", ERR_LIB_SSL, 188}, #endif + #ifdef SSL_R_NO_PEM_EXTENSIONS + {"NO_PEM_EXTENSIONS", ERR_LIB_SSL, SSL_R_NO_PEM_EXTENSIONS}, + #else + {"NO_PEM_EXTENSIONS", ERR_LIB_SSL, 389}, + #endif #ifdef SSL_R_NO_PRIVATEKEY {"NO_PRIVATEKEY", ERR_LIB_SSL, SSL_R_NO_PRIVATEKEY}, #else @@ -854,6 +999,16 @@ #else {"NO_SHARED_CIPHER", ERR_LIB_SSL, 193}, #endif + #ifdef SSL_R_NO_SHARED_SIGATURE_ALGORITHMS + {"NO_SHARED_SIGATURE_ALGORITHMS", ERR_LIB_SSL, SSL_R_NO_SHARED_SIGATURE_ALGORITHMS}, + #else + {"NO_SHARED_SIGATURE_ALGORITHMS", ERR_LIB_SSL, 376}, + #endif + #ifdef SSL_R_NO_SRTP_PROFILES + {"NO_SRTP_PROFILES", ERR_LIB_SSL, SSL_R_NO_SRTP_PROFILES}, + #else + {"NO_SRTP_PROFILES", ERR_LIB_SSL, 359}, + #endif #ifdef SSL_R_NO_VERIFY_CALLBACK {"NO_VERIFY_CALLBACK", ERR_LIB_SSL, SSL_R_NO_VERIFY_CALLBACK}, #else @@ -879,6 +1034,16 @@ #else {"OLD_SESSION_COMPRESSION_ALGORITHM_NOT_RETURNED", ERR_LIB_SSL, 344}, #endif + #ifdef SSL_R_ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE + {"ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE}, + #else + {"ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, 387}, + #endif + #ifdef SSL_R_ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE + {"ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE}, + #else + {"ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, 379}, + #endif #ifdef SSL_R_ONLY_TLS_ALLOWED_IN_FIPS_MODE {"ONLY_TLS_ALLOWED_IN_FIPS_MODE", ERR_LIB_SSL, SSL_R_ONLY_TLS_ALLOWED_IN_FIPS_MODE}, #else @@ -934,6 +1099,16 @@ #else {"PEER_ERROR_UNSUPPORTED_CERTIFICATE_TYPE", ERR_LIB_SSL, 204}, #endif + #ifdef SSL_R_PEM_NAME_BAD_PREFIX + {"PEM_NAME_BAD_PREFIX", ERR_LIB_SSL, SSL_R_PEM_NAME_BAD_PREFIX}, + #else + {"PEM_NAME_BAD_PREFIX", ERR_LIB_SSL, 391}, + #endif + #ifdef SSL_R_PEM_NAME_TOO_SHORT + {"PEM_NAME_TOO_SHORT", ERR_LIB_SSL, SSL_R_PEM_NAME_TOO_SHORT}, + #else + {"PEM_NAME_TOO_SHORT", ERR_LIB_SSL, 392}, + #endif #ifdef SSL_R_PRE_MAC_LENGTH_TOO_LONG {"PRE_MAC_LENGTH_TOO_LONG", ERR_LIB_SSL, SSL_R_PRE_MAC_LENGTH_TOO_LONG}, #else @@ -1069,11 +1244,36 @@ #else {"SHORT_READ", ERR_LIB_SSL, 219}, #endif + #ifdef SSL_R_SIGNATURE_ALGORITHMS_ERROR + {"SIGNATURE_ALGORITHMS_ERROR", ERR_LIB_SSL, SSL_R_SIGNATURE_ALGORITHMS_ERROR}, + #else + {"SIGNATURE_ALGORITHMS_ERROR", ERR_LIB_SSL, 360}, + #endif #ifdef SSL_R_SIGNATURE_FOR_NON_SIGNING_CERTIFICATE {"SIGNATURE_FOR_NON_SIGNING_CERTIFICATE", ERR_LIB_SSL, SSL_R_SIGNATURE_FOR_NON_SIGNING_CERTIFICATE}, #else {"SIGNATURE_FOR_NON_SIGNING_CERTIFICATE", ERR_LIB_SSL, 220}, #endif + #ifdef SSL_R_SRP_A_CALC + {"SRP_A_CALC", ERR_LIB_SSL, SSL_R_SRP_A_CALC}, + #else + {"SRP_A_CALC", ERR_LIB_SSL, 361}, + #endif + #ifdef SSL_R_SRTP_COULD_NOT_ALLOCATE_PROFILES + {"SRTP_COULD_NOT_ALLOCATE_PROFILES", ERR_LIB_SSL, SSL_R_SRTP_COULD_NOT_ALLOCATE_PROFILES}, + #else + {"SRTP_COULD_NOT_ALLOCATE_PROFILES", ERR_LIB_SSL, 362}, + #endif + #ifdef SSL_R_SRTP_PROTECTION_PROFILE_LIST_TOO_LONG + {"SRTP_PROTECTION_PROFILE_LIST_TOO_LONG", ERR_LIB_SSL, SSL_R_SRTP_PROTECTION_PROFILE_LIST_TOO_LONG}, + #else + {"SRTP_PROTECTION_PROFILE_LIST_TOO_LONG", ERR_LIB_SSL, 363}, + #endif + #ifdef SSL_R_SRTP_UNKNOWN_PROTECTION_PROFILE + {"SRTP_UNKNOWN_PROTECTION_PROFILE", ERR_LIB_SSL, SSL_R_SRTP_UNKNOWN_PROTECTION_PROFILE}, + #else + {"SRTP_UNKNOWN_PROTECTION_PROFILE", ERR_LIB_SSL, 364}, + #endif #ifdef SSL_R_SSL23_DOING_SESSION_ID_REUSE {"SSL23_DOING_SESSION_ID_REUSE", ERR_LIB_SSL, SSL_R_SSL23_DOING_SESSION_ID_REUSE}, #else @@ -1179,6 +1379,11 @@ #else {"SSL_LIBRARY_HAS_NO_CIPHERS", ERR_LIB_SSL, 230}, #endif + #ifdef SSL_R_SSL_NEGATIVE_LENGTH + {"SSL_NEGATIVE_LENGTH", ERR_LIB_SSL, SSL_R_SSL_NEGATIVE_LENGTH}, + #else + {"SSL_NEGATIVE_LENGTH", ERR_LIB_SSL, 372}, + #endif #ifdef SSL_R_SSL_SESSION_ID_CALLBACK_FAILED {"SSL_SESSION_ID_CALLBACK_FAILED", ERR_LIB_SSL, SSL_R_SSL_SESSION_ID_CALLBACK_FAILED}, #else @@ -1229,6 +1434,11 @@ #else {"TLSV1_ALERT_EXPORT_RESTRICTION", ERR_LIB_SSL, 1060}, #endif + #ifdef SSL_R_TLSV1_ALERT_INAPPROPRIATE_FALLBACK + {"TLSV1_ALERT_INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, SSL_R_TLSV1_ALERT_INAPPROPRIATE_FALLBACK}, + #else + {"TLSV1_ALERT_INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, 1086}, + #endif #ifdef SSL_R_TLSV1_ALERT_INSUFFICIENT_SECURITY {"TLSV1_ALERT_INSUFFICIENT_SECURITY", ERR_LIB_SSL, SSL_R_TLSV1_ALERT_INSUFFICIENT_SECURITY}, #else @@ -1294,6 +1504,21 @@ #else {"TLS_CLIENT_CERT_REQ_WITH_ANON_CIPHER", ERR_LIB_SSL, 232}, #endif + #ifdef SSL_R_TLS_HEARTBEAT_PEER_DOESNT_ACCEPT + {"TLS_HEARTBEAT_PEER_DOESNT_ACCEPT", ERR_LIB_SSL, SSL_R_TLS_HEARTBEAT_PEER_DOESNT_ACCEPT}, + #else + {"TLS_HEARTBEAT_PEER_DOESNT_ACCEPT", ERR_LIB_SSL, 365}, + #endif + #ifdef SSL_R_TLS_HEARTBEAT_PENDING + {"TLS_HEARTBEAT_PENDING", ERR_LIB_SSL, SSL_R_TLS_HEARTBEAT_PENDING}, + #else + {"TLS_HEARTBEAT_PENDING", ERR_LIB_SSL, 366}, + #endif + #ifdef SSL_R_TLS_ILLEGAL_EXPORTER_LABEL + {"TLS_ILLEGAL_EXPORTER_LABEL", ERR_LIB_SSL, SSL_R_TLS_ILLEGAL_EXPORTER_LABEL}, + #else + {"TLS_ILLEGAL_EXPORTER_LABEL", ERR_LIB_SSL, 367}, + #endif #ifdef SSL_R_TLS_INVALID_ECPOINTFORMAT_LIST {"TLS_INVALID_ECPOINTFORMAT_LIST", ERR_LIB_SSL, SSL_R_TLS_INVALID_ECPOINTFORMAT_LIST}, #else @@ -1399,6 +1624,16 @@ #else {"UNKNOWN_CIPHER_TYPE", ERR_LIB_SSL, 249}, #endif + #ifdef SSL_R_UNKNOWN_CMD_NAME + {"UNKNOWN_CMD_NAME", ERR_LIB_SSL, SSL_R_UNKNOWN_CMD_NAME}, + #else + {"UNKNOWN_CMD_NAME", ERR_LIB_SSL, 386}, + #endif + #ifdef SSL_R_UNKNOWN_DIGEST + {"UNKNOWN_DIGEST", ERR_LIB_SSL, SSL_R_UNKNOWN_DIGEST}, + #else + {"UNKNOWN_DIGEST", ERR_LIB_SSL, 368}, + #endif #ifdef SSL_R_UNKNOWN_KEY_EXCHANGE_TYPE {"UNKNOWN_KEY_EXCHANGE_TYPE", ERR_LIB_SSL, SSL_R_UNKNOWN_KEY_EXCHANGE_TYPE}, #else @@ -1469,16 +1704,36 @@ #else {"UNSUPPORTED_STATUS_TYPE", ERR_LIB_SSL, 329}, #endif + #ifdef SSL_R_USE_SRTP_NOT_NEGOTIATED + {"USE_SRTP_NOT_NEGOTIATED", ERR_LIB_SSL, SSL_R_USE_SRTP_NOT_NEGOTIATED}, + #else + {"USE_SRTP_NOT_NEGOTIATED", ERR_LIB_SSL, 369}, + #endif + #ifdef SSL_R_VERSION_TOO_LOW + {"VERSION_TOO_LOW", ERR_LIB_SSL, SSL_R_VERSION_TOO_LOW}, + #else + {"VERSION_TOO_LOW", ERR_LIB_SSL, 396}, + #endif #ifdef SSL_R_WRITE_BIO_NOT_SET {"WRITE_BIO_NOT_SET", ERR_LIB_SSL, SSL_R_WRITE_BIO_NOT_SET}, #else {"WRITE_BIO_NOT_SET", ERR_LIB_SSL, 260}, #endif + #ifdef SSL_R_WRONG_CERTIFICATE_TYPE + {"WRONG_CERTIFICATE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_CERTIFICATE_TYPE}, + #else + {"WRONG_CERTIFICATE_TYPE", ERR_LIB_SSL, 383}, + #endif #ifdef SSL_R_WRONG_CIPHER_RETURNED {"WRONG_CIPHER_RETURNED", ERR_LIB_SSL, SSL_R_WRONG_CIPHER_RETURNED}, #else {"WRONG_CIPHER_RETURNED", ERR_LIB_SSL, 261}, #endif + #ifdef SSL_R_WRONG_CURVE + {"WRONG_CURVE", ERR_LIB_SSL, SSL_R_WRONG_CURVE}, + #else + {"WRONG_CURVE", ERR_LIB_SSL, 378}, + #endif #ifdef SSL_R_WRONG_MESSAGE_TYPE {"WRONG_MESSAGE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_MESSAGE_TYPE}, #else @@ -1499,6 +1754,11 @@ #else {"WRONG_SIGNATURE_SIZE", ERR_LIB_SSL, 265}, #endif + #ifdef SSL_R_WRONG_SIGNATURE_TYPE + {"WRONG_SIGNATURE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_SIGNATURE_TYPE}, + #else + {"WRONG_SIGNATURE_TYPE", ERR_LIB_SSL, 370}, + #endif #ifdef SSL_R_WRONG_SSL_VERSION {"WRONG_SSL_VERSION", ERR_LIB_SSL, SSL_R_WRONG_SSL_VERSION}, #else @@ -1519,6 +1779,11 @@ #else {"X509_VERIFICATION_SETUP_PROBLEMS", ERR_LIB_SSL, 269}, #endif + #ifdef X509_R_AKID_MISMATCH + {"AKID_MISMATCH", ERR_LIB_X509, X509_R_AKID_MISMATCH}, + #else + {"AKID_MISMATCH", ERR_LIB_X509, 110}, + #endif #ifdef X509_R_BAD_X509_FILETYPE {"BAD_X509_FILETYPE", ERR_LIB_X509, X509_R_BAD_X509_FILETYPE}, #else @@ -1539,11 +1804,26 @@ #else {"CERT_ALREADY_IN_HASH_TABLE", ERR_LIB_X509, 101}, #endif + #ifdef X509_R_CRL_ALREADY_DELTA + {"CRL_ALREADY_DELTA", ERR_LIB_X509, X509_R_CRL_ALREADY_DELTA}, + #else + {"CRL_ALREADY_DELTA", ERR_LIB_X509, 127}, + #endif + #ifdef X509_R_CRL_VERIFY_FAILURE + {"CRL_VERIFY_FAILURE", ERR_LIB_X509, X509_R_CRL_VERIFY_FAILURE}, + #else + {"CRL_VERIFY_FAILURE", ERR_LIB_X509, 131}, + #endif #ifdef X509_R_ERR_ASN1_LIB {"ERR_ASN1_LIB", ERR_LIB_X509, X509_R_ERR_ASN1_LIB}, #else {"ERR_ASN1_LIB", ERR_LIB_X509, 102}, #endif + #ifdef X509_R_IDP_MISMATCH + {"IDP_MISMATCH", ERR_LIB_X509, X509_R_IDP_MISMATCH}, + #else + {"IDP_MISMATCH", ERR_LIB_X509, 128}, + #endif #ifdef X509_R_INVALID_DIRECTORY {"INVALID_DIRECTORY", ERR_LIB_X509, X509_R_INVALID_DIRECTORY}, #else @@ -1559,6 +1839,11 @@ #else {"INVALID_TRUST", ERR_LIB_X509, 123}, #endif + #ifdef X509_R_ISSUER_MISMATCH + {"ISSUER_MISMATCH", ERR_LIB_X509, X509_R_ISSUER_MISMATCH}, + #else + {"ISSUER_MISMATCH", ERR_LIB_X509, 129}, + #endif #ifdef X509_R_KEY_TYPE_MISMATCH {"KEY_TYPE_MISMATCH", ERR_LIB_X509, X509_R_KEY_TYPE_MISMATCH}, #else @@ -1584,11 +1869,21 @@ #else {"METHOD_NOT_SUPPORTED", ERR_LIB_X509, 124}, #endif + #ifdef X509_R_NEWER_CRL_NOT_NEWER + {"NEWER_CRL_NOT_NEWER", ERR_LIB_X509, X509_R_NEWER_CRL_NOT_NEWER}, + #else + {"NEWER_CRL_NOT_NEWER", ERR_LIB_X509, 132}, + #endif #ifdef X509_R_NO_CERT_SET_FOR_US_TO_VERIFY {"NO_CERT_SET_FOR_US_TO_VERIFY", ERR_LIB_X509, X509_R_NO_CERT_SET_FOR_US_TO_VERIFY}, #else {"NO_CERT_SET_FOR_US_TO_VERIFY", ERR_LIB_X509, 105}, #endif + #ifdef X509_R_NO_CRL_NUMBER + {"NO_CRL_NUMBER", ERR_LIB_X509, X509_R_NO_CRL_NUMBER}, + #else + {"NO_CRL_NUMBER", ERR_LIB_X509, 130}, + #endif #ifdef X509_R_PUBLIC_KEY_DECODE_ERROR {"PUBLIC_KEY_DECODE_ERROR", ERR_LIB_X509, X509_R_PUBLIC_KEY_DECODE_ERROR}, #else diff --git a/Tools/ssl/make_ssl_data.py b/Tools/ssl/make_ssl_data.py --- a/Tools/ssl/make_ssl_data.py +++ b/Tools/ssl/make_ssl_data.py @@ -5,8 +5,7 @@ `library` and `reason` mnemnonics to a more recent OpenSSL version. It takes two arguments: -- the path to the OpenSSL include files' directory - (e.g. openssl-1.0.1-beta3/include/openssl/) +- the path to the OpenSSL source tree (e.g. git checkout) - the path to the C file to be generated (probably Modules/_ssl_data.h) """ @@ -15,9 +14,10 @@ import os import re import sys +import _ssl -def parse_error_codes(h_file, prefix): +def parse_error_codes(h_file, prefix, libcode): pat = re.compile(r"#define\W+(%s([\w]+))\W+(\d+)\b" % re.escape(prefix)) codes = [] with open(h_file, "r", encoding="latin1") as f: @@ -26,7 +26,8 @@ if match: code, name, num = match.groups() num = int(num) - codes.append((code, name, num)) + # e.g. ("SSL_R_BAD_DATA", ("ERR_LIB_SSL", "BAD_DATA", 390)) + codes.append((code, (libcode, name, num))) return codes if __name__ == "__main__": @@ -34,12 +35,32 @@ outfile = sys.argv[2] use_stdout = outfile == '-' f = sys.stdout if use_stdout else open(outfile, "w") - error_libraries = ( - # (library code, mnemonic, error prefix, header file) - ('ERR_LIB_PEM', 'PEM', 'PEM_R_', 'pem.h'), - ('ERR_LIB_SSL', 'SSL', 'SSL_R_', 'ssl.h'), - ('ERR_LIB_X509', 'X509', 'X509_R_', 'x509.h'), - ) + error_libraries = { + # mnemonic -> (library code, error prefix, header file) + 'PEM': ('ERR_LIB_PEM', 'PEM_R_', 'crypto/pem/pem.h'), + 'SSL': ('ERR_LIB_SSL', 'SSL_R_', 'ssl/ssl.h'), + 'X509': ('ERR_LIB_X509', 'X509_R_', 'crypto/x509/x509.h'), + } + + # Read codes from libraries + new_codes = [] + for libcode, prefix, h_file in sorted(error_libraries.values()): + new_codes += parse_error_codes(os.path.join(openssl_inc, h_file), + prefix, libcode) + new_code_nums = set((libcode, num) + for (code, (libcode, name, num)) in new_codes) + + # Merge with existing codes (in case some old codes disappeared). + codes = {} + for errname, (libnum, errnum) in _ssl.err_names_to_codes.items(): + lib = error_libraries[_ssl.lib_codes_to_names[libnum]] + libcode = lib[0] # e.g. ERR_LIB_PEM + errcode = lib[1] + errname # e.g. SSL_R_BAD_SSL_SESSION_ID_LENGTH + # Only keep it if the numeric codes weren't reused + if (libcode, errnum) not in new_code_nums: + codes[errcode] = libcode, errname, errnum + codes.update(dict(new_codes)) + def w(l): f.write(l + "\n") w("/* File generated by Tools/ssl/make_ssl_data.py */") @@ -47,21 +68,19 @@ w("") w("static struct py_ssl_library_code library_codes[] = {") - for libcode, mnemo, _, _ in error_libraries: + for mnemo, (libcode, _, _) in sorted(error_libraries.items()): w(' {"%s", %s},' % (mnemo, libcode)) w(' { NULL }') w('};') w("") w("static struct py_ssl_error_code error_codes[] = {") - for libcode, _, prefix, h_file in error_libraries: - codes = parse_error_codes(os.path.join(openssl_inc, h_file), prefix) - for code, name, num in sorted(codes): - w(' #ifdef %s' % (code)) - w(' {"%s", %s, %s},' % (name, libcode, code)) - w(' #else') - w(' {"%s", %s, %d},' % (name, libcode, num)) - w(' #endif') + for errcode, (libcode, name, num) in sorted(codes.items()): + w(' #ifdef %s' % (errcode)) + w(' {"%s", %s, %s},' % (name, libcode, errcode)) + w(' #else') + w(' {"%s", %s, %d},' % (name, libcode, num)) + w(' #endif') w(' { NULL }') w('};') if not use_stdout: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 21:37:26 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 20:37:26 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323133=3A_Pickling?= =?utf-8?q?_of_ipaddress_objects_now_produces_more_compact_and?= Message-ID: <20150118203722.93177.15604@psf.io> https://hg.python.org/cpython/rev/781b54f7bccc changeset: 94212:781b54f7bccc parent: 94210:15f46b850257 user: Serhiy Storchaka date: Sun Jan 18 22:36:33 2015 +0200 summary: Issue #23133: Pickling of ipaddress objects now produces more compact and portable representation. files: Lib/ipaddress.py | 21 +++++++++++----- Lib/test/test_ipaddress.py | 31 ++++++++++++++++++++----- Misc/NEWS | 3 ++ 3 files changed, 41 insertions(+), 14 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -567,6 +567,9 @@ except ValueError: cls._report_invalid_netmask(ip_str) + def __reduce__(self): + return self.__class__, (str(self),) + class _BaseAddress(_IPAddressBase): @@ -576,11 +579,6 @@ used by single IP addresses. """ - def __init__(self, address): - if (not isinstance(address, bytes) - and '/' in str(address)): - raise AddressValueError("Unexpected '/' in %r" % address) - def __int__(self): return self._ip @@ -626,6 +624,9 @@ def _get_address_key(self): return (self._version, self) + def __reduce__(self): + return self.__class__, (self._ip,) + class _BaseNetwork(_IPAddressBase): @@ -1295,7 +1296,6 @@ AddressValueError: If ipaddress isn't a valid IPv4 address. """ - _BaseAddress.__init__(self, address) _BaseV4.__init__(self, address) # Efficient constructor from integer. @@ -1313,6 +1313,8 @@ # Assume input argument to be string or any object representation # which converts into a formatted IP string. addr_str = str(address) + if '/' in addr_str: + raise AddressValueError("Unexpected '/' in %r" % address) self._ip = self._ip_int_from_string(addr_str) @property @@ -1446,6 +1448,8 @@ def __hash__(self): return self._ip ^ self._prefixlen ^ int(self.network.network_address) + __reduce__ = _IPAddressBase.__reduce__ + @property def ip(self): return IPv4Address(self._ip) @@ -1920,7 +1924,6 @@ AddressValueError: If address isn't a valid IPv6 address. """ - _BaseAddress.__init__(self, address) _BaseV6.__init__(self, address) # Efficient constructor from integer. @@ -1938,6 +1941,8 @@ # Assume input argument to be string or any object representation # which converts into a formatted IP string. addr_str = str(address) + if '/' in addr_str: + raise AddressValueError("Unexpected '/' in %r" % address) self._ip = self._ip_int_from_string(addr_str) @property @@ -2134,6 +2139,8 @@ def __hash__(self): return self._ip ^ self._prefixlen ^ int(self.network.network_address) + __reduce__ = _IPAddressBase.__reduce__ + @property def ip(self): return IPv6Address(self._ip) diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -8,6 +8,7 @@ import re import contextlib import operator +import pickle import ipaddress @@ -82,6 +83,13 @@ self.assertRaises(TypeError, hex, self.factory(1)) self.assertRaises(TypeError, bytes, self.factory(1)) + def pickle_test(self, addr): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + x = self.factory(addr) + y = pickle.loads(pickle.dumps(x, proto)) + self.assertEqual(y, x) + class CommonTestMixin_v4(CommonTestMixin): @@ -247,6 +255,9 @@ assertBadOctet("257.0.0.0", 257) assertBadOctet("192.168.0.999", 999) + def test_pickle(self): + self.pickle_test('192.0.2.1') + class AddressTestCase_v6(BaseTestCase, CommonTestMixin_v6): factory = ipaddress.IPv6Address @@ -379,6 +390,9 @@ assertBadPart("02001:db8::", "02001") assertBadPart('2001:888888::1', "888888") + def test_pickle(self): + self.pickle_test('2001:db8::') + class NetmaskTestMixin_v4(CommonTestMixin_v4): """Input validation on interfaces and networks is very similar""" @@ -446,6 +460,11 @@ class InterfaceTestCase_v4(BaseTestCase, NetmaskTestMixin_v4): factory = ipaddress.IPv4Interface + def test_pickle(self): + self.pickle_test('192.0.2.0/27') + self.pickle_test('192.0.2.0/31') # IPV4LENGTH - 1 + self.pickle_test('192.0.2.0') # IPV4LENGTH + class NetworkTestCase_v4(BaseTestCase, NetmaskTestMixin_v4): factory = ipaddress.IPv4Network @@ -500,6 +519,11 @@ assertBadNetmask("::1", "pudding") assertBadNetmask("::", "::") + def test_pickle(self): + self.pickle_test('2001:db8::1000/124') + self.pickle_test('2001:db8::1000/127') # IPV6LENGTH - 1 + self.pickle_test('2001:db8::1000') # IPV6LENGTH + class InterfaceTestCase_v6(BaseTestCase, NetmaskTestMixin_v6): factory = ipaddress.IPv6Interface @@ -774,13 +798,6 @@ self.assertEqual(128, ipaddress._count_righthand_zero_bits(0, 128)) self.assertEqual("IPv4Network('1.2.3.0/24')", repr(self.ipv4_network)) - def testMissingAddressVersion(self): - class Broken(ipaddress._BaseAddress): - pass - broken = Broken('127.0.0.1') - with self.assertRaisesRegex(NotImplementedError, "Broken.*version"): - broken.version - def testMissingNetworkVersion(self): class Broken(ipaddress._BaseNetwork): pass diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,9 @@ Library ------- +- Issue #23133: Pickling of ipaddress objects now produces more compact and + portable representation. + - Issue #23248: Update ssl error codes from latest OpenSSL git master. - Issue #23266: Much faster implementation of ipaddress.collapse_addresses() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 21:57:31 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 20:57:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fixed_tests_for_issue_=232?= =?utf-8?q?3133_=28pickling_of_IPv4Network_was_not_tested=29=2E?= Message-ID: <20150118205728.104126.29879@psf.io> https://hg.python.org/cpython/rev/712ac77b772b changeset: 94213:712ac77b772b user: Serhiy Storchaka date: Sun Jan 18 22:56:47 2015 +0200 summary: Fixed tests for issue #23133 (pickling of IPv4Network was not tested). files: Lib/test/test_ipaddress.py | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -456,16 +456,16 @@ assertBadNetmask("1.1.1.1", "pudding") assertBadNetmask("1.1.1.1", "::") - -class InterfaceTestCase_v4(BaseTestCase, NetmaskTestMixin_v4): - factory = ipaddress.IPv4Interface - def test_pickle(self): self.pickle_test('192.0.2.0/27') self.pickle_test('192.0.2.0/31') # IPV4LENGTH - 1 self.pickle_test('192.0.2.0') # IPV4LENGTH +class InterfaceTestCase_v4(BaseTestCase, NetmaskTestMixin_v4): + factory = ipaddress.IPv4Interface + + class NetworkTestCase_v4(BaseTestCase, NetmaskTestMixin_v4): factory = ipaddress.IPv4Network -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 22:13:03 2015 From: python-checkins at python.org (raymond.hettinger) Date: Sun, 18 Jan 2015 21:13:03 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_23261=3A__Clean-up_t?= =?utf-8?q?he_hack_to_store_the_set=2Epop=28=29_search_finger_in_a_hash?= Message-ID: <20150118211249.69922.67328@psf.io> https://hg.python.org/cpython/rev/0f90a3bd07f7 changeset: 94214:0f90a3bd07f7 user: Raymond Hettinger date: Sun Jan 18 13:12:42 2015 -0800 summary: Issue 23261: Clean-up the hack to store the set.pop() search finger in a hash field instead of the setobject. files: Include/setobject.h | 5 +-- Lib/test/test_sys.py | 2 +- Objects/setobject.c | 33 +++++++++++-------------------- 3 files changed, 15 insertions(+), 25 deletions(-) diff --git a/Include/setobject.h b/Include/setobject.h --- a/Include/setobject.h +++ b/Include/setobject.h @@ -14,9 +14,7 @@ 2. Active: key != NULL and key != dummy 3. Dummy: key == dummy -Note: .pop() abuses the hash field of an Unused or Dummy slot to -hold a search finger. The hash field of Unused or Dummy slots has -no meaning otherwise. +The hash field of Unused or Dummy slots have no meaning. */ #define PySet_MINSIZE 8 @@ -59,6 +57,7 @@ Py_hash_t hash; /* Only used by frozenset objects */ setentry smalltable[PySet_MINSIZE]; + Py_ssize_t finger; /* Search finger for pop() */ PyObject *weakreflist; /* List of weak references */ } PySetObject; diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -994,7 +994,7 @@ # frozenset PySet_MINSIZE = 8 samples = [[], range(10), range(50)] - s = size('3n2P' + PySet_MINSIZE*'nP' + 'nP') + s = size('3n2P' + PySet_MINSIZE*'nP' + '2nP') for sample in samples: minused = len(sample) if minused == 0: tmp = 1 diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -668,32 +668,22 @@ return NULL; } - /* Set entry to "the first" unused or dummy set entry. We abuse - * the hash field of slot 0 to hold a search finger: - * If slot 0 has a value, use slot 0. - * Else slot 0 is being used to hold a search finger, - * and we use its hash value as the first index to look. + i = so->finger; + /* This may be a legit search finger, or it may be a once legit + * search finger that's out of bounds now (due to wrapping or + * resizing). We simply make sure it's in bounds now. */ - entry = &so->table[0]; - if (entry->key == NULL || entry->key == dummy) { - i = entry->hash; - /* The hash field may be a real hash value, or it may be a - * legit search finger, or it may be a once-legit search - * finger that's out of bounds now because it wrapped around - * or the table shrunk -- simply make sure it's in bounds now. - */ - if (i > so->mask || i < 1) - i = 1; /* skip slot 0 */ - while ((entry = &so->table[i])->key == NULL || entry->key==dummy) { - i++; - if (i > so->mask) - i = 1; - } + if (i > so->mask) + i = 0; + while ((entry = &so->table[i])->key == NULL || entry->key==dummy) { + i++; + if (i > so->mask) + i = 0; } key = entry->key; entry->key = dummy; so->used--; - so->table[0].hash = i + 1; /* next place to start */ + so->finger = i + 1; /* next place to start */ return key; } @@ -1012,6 +1002,7 @@ so->table = so->smalltable; so->lookup = set_lookkey_unicode; so->hash = -1; + so->finger = 0; so->weakreflist = NULL; if (iterable != NULL) { -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 18 23:42:18 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 18 Jan 2015 22:42:18 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323266=3A_Restore_?= =?utf-8?q?the_performance_of_ipaddress=2Ecollapse=5Faddresses=28=29_whith?= Message-ID: <20150118224218.104122.93102@psf.io> https://hg.python.org/cpython/rev/021b23a40f9f changeset: 94215:021b23a40f9f user: Serhiy Storchaka date: Mon Jan 19 00:41:32 2015 +0200 summary: Issue #23266: Restore the performance of ipaddress.collapse_addresses() whith duplicated addresses and simplify the code. files: Lib/ipaddress.py | 36 +++++++++++-------------- Lib/test/test_ipaddress.py | 14 ++++++--- 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -164,25 +164,23 @@ def _find_address_range(addresses): - """Find a sequence of IPv#Address. + """Find a sequence of sorted deduplicated IPv#Address. Args: addresses: a list of IPv#Address objects. - Returns: - A tuple containing the first and last IP addresses in the sequence, - and the number of distinct IP addresses in the sequence. + Yields: + A tuple containing the first and last IP addresses in the sequence. """ - first = last = addresses[0] - i = 1 - for ip in addresses[1:]: - if ip._ip == last._ip + 1: - last = ip - i += 1 - else: - break - return (first, last, i) + it = iter(addresses) + first = last = next(it) + for ip in it: + if ip._ip != last._ip + 1: + yield first, last + first = ip + last = ip + yield first, last def _count_righthand_zero_bits(number, bits): @@ -323,7 +321,6 @@ TypeError: If passed a list of mixed version objects. """ - i = 0 addrs = [] ips = [] nets = [] @@ -349,14 +346,13 @@ ip, nets[-1])) nets.append(ip) - # sort - ips = sorted(ips) + # sort and dedup + ips = sorted(set(ips)) # find consecutive address ranges in the sorted sequence and summarize them - while i < len(ips): - (first, last, items) = _find_address_range(ips[i:]) - i = items + i - addrs.extend(summarize_address_range(first, last)) + if ips: + for first, last in _find_address_range(ips): + addrs.extend(summarize_address_range(first, last)) return _collapse_addresses_internal(addrs + nets) diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -790,11 +790,15 @@ 2 ** ipaddress.IPV6LENGTH) def testInternals(self): - first, last, nitems = ipaddress._find_address_range([ - ipaddress.IPv4Address('10.10.10.10'), - ipaddress.IPv4Address('10.10.10.12')]) - self.assertEqual(first, last) - self.assertEqual(nitems, 1) + ip1 = ipaddress.IPv4Address('10.10.10.10') + ip2 = ipaddress.IPv4Address('10.10.10.11') + ip3 = ipaddress.IPv4Address('10.10.10.12') + self.assertEqual(list(ipaddress._find_address_range([ip1])), + [(ip1, ip1)]) + self.assertEqual(list(ipaddress._find_address_range([ip1, ip3])), + [(ip1, ip1), (ip3, ip3)]) + self.assertEqual(list(ipaddress._find_address_range([ip1, ip2, ip3])), + [(ip1, ip3)]) self.assertEqual(128, ipaddress._count_righthand_zero_bits(0, 128)) self.assertEqual("IPv4Network('1.2.3.0/24')", repr(self.ipv4_network)) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 19 01:08:12 2015 From: python-checkins at python.org (raymond.hettinger) Date: Mon, 19 Jan 2015 00:08:12 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Clean-up=2C_simplify=2C_an?= =?utf-8?q?d_slightly_speed-up_bounds_logic_in_set=5Fpop=28=29=2E?= Message-ID: <20150119000625.103315.64694@psf.io> https://hg.python.org/cpython/rev/e110b6263951 changeset: 94216:e110b6263951 user: Raymond Hettinger date: Sun Jan 18 16:06:18 2015 -0800 summary: Clean-up, simplify, and slightly speed-up bounds logic in set_pop(). Elsewhere in the setobject.c code we do a bitwise-and with the mask instead of using a conditional to reset to zero on wrap-around. Using that same technique here use gives cleaner, faster, and more consistent code. files: Objects/setobject.c | 13 +++---------- 1 files changed, 3 insertions(+), 10 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -658,7 +658,8 @@ static PyObject * set_pop(PySetObject *so) { - Py_ssize_t i = 0; + /* Make sure the search finger is in bounds */ + Py_ssize_t i = so->finger & so->mask; setentry *entry; PyObject *key; @@ -668,17 +669,9 @@ return NULL; } - i = so->finger; - /* This may be a legit search finger, or it may be a once legit - * search finger that's out of bounds now (due to wrapping or - * resizing). We simply make sure it's in bounds now. - */ - if (i > so->mask) - i = 0; while ((entry = &so->table[i])->key == NULL || entry->key==dummy) { i++; - if (i > so->mask) - i = 0; + i &= so->mask; } key = entry->key; entry->key = dummy; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 19 01:10:38 2015 From: python-checkins at python.org (raymond.hettinger) Date: Mon, 19 Jan 2015 00:10:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Update_copyright_for_2015_?= =?utf-8?q?updates=2E?= Message-ID: <20150119001035.118100.75340@psf.io> https://hg.python.org/cpython/rev/e1c8583c7e41 changeset: 94217:e1c8583c7e41 user: Raymond Hettinger date: Sun Jan 18 16:10:30 2015 -0800 summary: Update copyright for 2015 updates. files: Objects/setobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -4,7 +4,7 @@ Written and maintained by Raymond D. Hettinger Derived from Lib/sets.py and Objects/dictobject.c. - Copyright (c) 2003-2014 Python Software Foundation. + Copyright (c) 2003-2015 Python Software Foundation. All rights reserved. The basic lookup function used by all operations. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 19 06:25:24 2015 From: python-checkins at python.org (raymond.hettinger) Date: Mon, 19 Jan 2015 05:25:24 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_A_hybrid_of_and-masking_an?= =?utf-8?q?d_a_conditional-set-to-zero_produce_even_faster?= Message-ID: <20150119052521.104132.88198@psf.io> https://hg.python.org/cpython/rev/ac0d6c09457e changeset: 94218:ac0d6c09457e user: Raymond Hettinger date: Sun Jan 18 21:25:15 2015 -0800 summary: A hybrid of and-masking and a conditional-set-to-zero produce even faster search loop. files: Objects/setobject.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -671,7 +671,8 @@ while ((entry = &so->table[i])->key == NULL || entry->key==dummy) { i++; - i &= so->mask; + if (i > so->mask) + i = 0; } key = entry->key; entry->key = dummy; -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Mon Jan 19 09:26:14 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 19 Jan 2015 09:26:14 +0100 Subject: [Python-checkins] Daily reference leaks (e1c8583c7e41): sum=1 Message-ID: results for e1c8583c7e41 on branch "default" -------------------------------------------- test_collections leaked [2, -2, -2] references, sum=-2 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogUHsIR1', '-x'] From python-checkins at python.org Mon Jan 19 23:31:18 2015 From: python-checkins at python.org (chris.angelico) Date: Mon, 19 Jan 2015 22:31:18 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Remove_over-specified_text_ab?= =?utf-8?q?out_finding_PEP_source=2C_link_instead_to_repo?= Message-ID: <20150119223017.118080.70010@psf.io> https://hg.python.org/peps/rev/b93444aa3d3b changeset: 5675:b93444aa3d3b user: Chris Angelico date: Tue Jan 20 09:30:07 2015 +1100 summary: Remove over-specified text about finding PEP source, link instead to repo files: pep-0012.txt | 5 +- pep-0479.txt.orig | 603 ++++++++++++++++++++++++++++++++++ pep-0483.txt.orig | 328 ++++++++++++++++++ 3 files changed, 933 insertions(+), 3 deletions(-) diff --git a/pep-0012.txt b/pep-0012.txt --- a/pep-0012.txt +++ b/pep-0012.txt @@ -23,9 +23,8 @@ the text (reStructuredText) source of this PEP in order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR TEMPLATE!** -To get the source this (or any) PEP, look at the top of the HTML page -and click on the date & time on the "Last-Modified" line. It is a -link to the source text in the Python repository. +The source for this (or any) PEP can be found in the PEPs repository, +viewable on the web at https://hg.python.org/peps/file/tip . If you would prefer not to use markup in your PEP, please see PEP 9, "Sample Plaintext PEP Template" [2]_. diff --git a/pep-0479.txt.orig b/pep-0479.txt.orig new file mode 100644 --- /dev/null +++ b/pep-0479.txt.orig @@ -0,0 +1,603 @@ +PEP: 479 +Title: Change StopIteration handling inside generators +Version: $Revision$ +Last-Modified: $Date$ +Author: Chris Angelico , Guido van Rossum +Status: Accepted +Type: Standards Track +Content-Type: text/x-rst +Created: 15-Nov-2014 +Python-Version: 3.5 +Post-History: 15-Nov-2014, 19-Nov-2014, 5-Dec-2014 + + +Abstract +======== + +This PEP proposes a change to generators: when ``StopIteration`` is +raised inside a generator, it is replaced it with ``RuntimeError``. +(More precisely, this happens when the exception is about to bubble +out of the generator's stack frame.) Because the change is backwards +incompatible, the feature is initially introduced using a +``__future__`` statement. + + +Acceptance +========== + +This PEP was accepted by the BDFL on November 22. Because of the +exceptionally short period from first draft to acceptance, the main +objections brought up after acceptance were carefully considered and +have been reflected in the "Alternate proposals" section below. +However, none of the discussion changed the BDFL's mind and the PEP's +acceptance is now final. (Suggestions for clarifying edits are still +welcome -- unlike IETF RFCs, the text of a PEP is not cast in stone +after its acceptance, although the core design/plan/specification +should not change after acceptance.) + + +Rationale +========= + +The interaction of generators and ``StopIteration`` is currently +somewhat surprising, and can conceal obscure bugs. An unexpected +exception should not result in subtly altered behaviour, but should +cause a noisy and easily-debugged traceback. Currently, +``StopIteration`` can be absorbed by the generator construct. + +The main goal of the proposal is to ease debugging in the situation +where an unguarded ``next()`` call (perhaps several stack frames deep) +raises ``StopIteration`` and causes the iteration controlled by the +generator to terminate silently. (When another exception is raised, a +traceback is printed pinpointing the cause of the problem.) + +This is particularly pernicious in combination with the ``yield from`` +construct of PEP 380 [1]_, as it breaks the abstraction that a +subgenerator may be factored out of a generator. That PEP notes this +limitation, but notes that "use cases for these [are] rare to non- +existent". Unfortunately while intentional use is rare, it is easy to +stumble on these cases by accident:: + + import contextlib + + @contextlib.contextmanager + def transaction(): + print('begin') + try: + yield from do_it() + except: + print('rollback') + raise + else: + print('commit') + + def do_it(): + print('Refactored initial setup') + yield # Body of with-statement is executed here + print('Refactored finalization of successful transaction') + + + import pathlib + + with transaction(): + print('commit file {}'.format( + # I can never remember what the README extension is + next(pathlib.Path('/some/dir').glob('README*')))) + +Here factoring out ``do_it`` into a subgenerator has introduced a subtle +bug: if a bug in the wrapped block allows ``StopIteration`` to escape +(here because the README doesn't exist), under the current behavior +``do_it`` will properly abort, but the exception will be swallowed by +the ``yield from`` and the original context manager will commit the +unfinished transaction! Similarly problematic behavior occurs when an +``asyncio`` coroutine raises ``StopIteration``, causing it to terminate +silently. In both cases, the refactoring abstraction of ``yield from`` +breaks in the presence of bugs in client code. + +Additionally, the proposal reduces the difference between list +comprehensions and generator expressions, preventing surprises such as +the one that started this discussion [2]_. Henceforth, the following +statements will produce the same result if either produces a result at +all:: + + a = list(F(x) for x in xs if P(x)) + a = [F(x) for x in xs if P(x)] + +With the current state of affairs, it is possible to write a function +``F(x)`` or a predicate ``P(x)`` that causes the first form to produce +a (truncated) result, while the second form raises an exception +(namely, ``StopIteration``). With the proposed change, both forms +will raise an exception at this point (albeit ``RuntimeError`` in the +first case and ``StopIteration`` in the second). + +Finally, the proposal also clears up the confusion about how to +terminate a generator: the proper way is ``return``, not +``raise StopIteration``. + +As an added bonus, the above changes bring generator functions much +more in line with regular functions. If you wish to take a piece of +code presented as a generator and turn it into something else, you +can usually do this fairly simply, by replacing every ``yield`` with +a call to ``print()`` or ``list.append()``; however, if there are any +bare ``next()`` calls in the code, you have to be aware of them. If +the code was originally written without relying on ``StopIteration`` +terminating the function, the transformation would be that much +easier. + + +Background information +====================== + +When a generator frame is (re)started as a result of a ``__next__()`` +(or ``send()`` or ``throw()``) call, one of three outcomes can occur: + +* A yield point is reached, and the yielded value is returned. +* The frame is returned from; ``StopIteration`` is raised. +* An exception is raised, which bubbles out. + +In the latter two cases the frame is abandoned (and the generator +object's ``gi_frame`` attribute is set to None). + + +Proposal +======== + +If a ``StopIteration`` is about to bubble out of a generator frame, it +is replaced with ``RuntimeError``, which causes the ``next()`` call +(which invoked the generator) to fail, passing that exception out. +From then on it's just like any old exception. [4]_ + +This affects the third outcome listed above, without altering any +other effects. Furthermore, it only affects this outcome when the +exception raised is ``StopIteration`` (or a subclass thereof). + +Note that the proposed replacement happens at the point where the +exception is about to bubble out of the frame, i.e. after any +``except`` or ``finally`` blocks that could affect it have been +exited. The ``StopIteration`` raised by returning from the frame is +not affected (the point being that ``StopIteration`` means that the +generator terminated "normally", i.e. it did not raise an exception). + +A subtle issue is what will happen if the caller, having caught the +``RuntimeError``, calls the generator object's ``__next__()`` method +again. The answer is that from this point on it will raise +``StopIteration`` -- the behavior is the same as when any other +exception was raised by the generator. + +Another logical consequence of the proposal: if someone uses +``g.throw(StopIteration)`` to throw a ``StopIteration`` exception into +a generator, if the generator doesn't catch it (which it could do +using a ``try/except`` around the ``yield``), it will be transformed +into ``RuntimeError``. + +During the transition phase, the new feature must be enabled +per-module using:: + + from __future__ import generator_stop + +Any generator function constructed under the influence of this +directive will have the ``REPLACE_STOPITERATION`` flag set on its code +object, and generators with the flag set will behave according to this +proposal. Once the feature becomes standard, the flag may be dropped; +code should not inspect generators for it. + + +Consequences for existing code +============================== + +This change will affect existing code that depends on +``StopIteration`` bubbling up. The pure Python reference +implementation of ``groupby`` [3]_ currently has comments "Exit on +``StopIteration``" where it is expected that the exception will +propagate and then be handled. This will be unusual, but not unknown, +and such constructs will fail. Other examples abound, e.g. [6]_, [7]_. + +(Nick Coghlan comments: """If you wanted to factor out a helper +function that terminated the generator you'd have to do "return +yield from helper()" rather than just "helper()".""") + +There are also examples of generator expressions floating around that +rely on a ``StopIteration`` raised by the expression, the target or the +predicate (rather than by the ``__next__()`` call implied in the ``for`` +loop proper). + +Writing backwards and forwards compatible code +---------------------------------------------- + +With the exception of hacks that raise ``StopIteration`` to exit a +generator expression, it is easy to write code that works equally well +under older Python versions as under the new semantics. + +This is done by enclosing those places in the generator body where a +``StopIteration`` is expected (e.g. bare ``next()`` calls or in some +cases helper functions that are expected to raise ``StopIteration``) +in a ``try/except`` construct that returns when ``StopIteration`` is +raised. The ``try/except`` construct should appear directly in the +generator function; doing this in a helper function that is not itself +a generator does not work. If ``raise StopIteration`` occurs directly +in a generator, simply replace it with ``return``. + + +Examples of breakage +-------------------- + +Generators which explicitly raise ``StopIteration`` can generally be +changed to simply return instead. This will be compatible with all +existing Python versions, and will not be affected by ``__future__``. +Here are some illustrations from the standard library. + +Lib/ipaddress.py:: + + if other == self: + raise StopIteration + +Becomes:: + + if other == self: + return + +In some cases, this can be combined with ``yield from`` to simplify +the code, such as Lib/difflib.py:: + + if context is None: + while True: + yield next(line_pair_iterator) + +Becomes:: + + if context is None: + yield from line_pair_iterator + return + +(The ``return`` is necessary for a strictly-equivalent translation, +though in this particular file, there is no further code, and the +``return`` can be omitted.) For compatibility with pre-3.3 versions +of Python, this could be written with an explicit ``for`` loop:: + + if context is None: + for line in line_pair_iterator: + yield line + return + +More complicated iteration patterns will need explicit ``try/except`` +constructs. For example, a hypothetical parser like this:: + + def parser(f): + while True: + data = next(f) + while True: + line = next(f) + if line == "- end -": break + data += line + yield data + +would need to be rewritten as:: + + def parser(f): + while True: + try: + data = next(f) + while True: + line = next(f) + if line == "- end -": break + data += line + yield data + except StopIteration: + return + +or possibly:: + + def parser(f): + for data in f: + while True: + line = next(f) + if line == "- end -": break + data += line + yield data + +The latter form obscures the iteration by purporting to iterate over +the file with a ``for`` loop, but then also fetches more data from +the same iterator during the loop body. It does, however, clearly +differentiate between a "normal" termination (``StopIteration`` +instead of the initial line) and an "abnormal" termination (failing +to find the end marker in the inner loop, which will now raise +``RuntimeError``). + +This effect of ``StopIteration`` has been used to cut a generator +expression short, creating a form of ``takewhile``:: + + def stop(): + raise StopIteration + print(list(x for x in range(10) if x < 5 or stop())) + # prints [0, 1, 2, 3, 4] + +Under the current proposal, this form of non-local flow control is +not supported, and would have to be rewritten in statement form:: + + def gen(): + for x in range(10): + if x >= 5: return + yield x + print(list(gen())) + # prints [0, 1, 2, 3, 4] + +While this is a small loss of functionality, it is functionality that +often comes at the cost of readability, and just as ``lambda`` has +restrictions compared to ``def``, so does a generator expression have +restrictions compared to a generator function. In many cases, the +transformation to full generator function will be trivially easy, and +may improve structural clarity. + + +Explanation of generators, iterators, and StopIteration +======================================================= + +Under this proposal, generators and iterators would be distinct, but +related, concepts. Like the mixing of text and bytes in Python 2, +the mixing of generators and iterators has resulted in certain +perceived conveniences, but proper separation will make bugs more +visible. + +An iterator is an object with a ``__next__`` method. Like many other +special methods, it may either return a value, or raise a specific +exception - in this case, ``StopIteration`` - to signal that it has +no value to return. In this, it is similar to ``__getattr__`` (can +raise ``AttributeError``), ``__getitem__`` (can raise ``KeyError``), +and so on. A helper function for an iterator can be written to +follow the same protocol; for example:: + + def helper(x, y): + if x > y: return 1 / (x - y) + raise StopIteration + + def __next__(self): + if self.a: return helper(self.b, self.c) + return helper(self.d, self.e) + +Both forms of signalling are carried through: a returned value is +returned, an exception bubbles up. The helper is written to match +the protocol of the calling function. + +A generator function is one which contains a ``yield`` expression. +Each time it is (re)started, it may either yield a value, or return +(including "falling off the end"). A helper function for a generator +can also be written, but it must also follow generator protocol:: + + def helper(x, y): + if x > y: yield 1 / (x - y) + + def gen(self): + if self.a: return (yield from helper(self.b, self.c)) + return (yield from helper(self.d, self.e)) + +In both cases, any unexpected exception will bubble up. Due to the +nature of generators and iterators, an unexpected ``StopIteration`` +inside a generator will be converted into ``RuntimeError``, but +beyond that, all exceptions will propagate normally. + + +Transition plan +=============== + +- Python 3.5: Enable new semantics under ``__future__`` import; silent + deprecation warning if ``StopIteration`` bubbles out of a generator + not under ``__future__`` import. + +- Python 3.6: Non-silent deprecation warning. + +- Python 3.7: Enable new semantics everywhere. + + +Alternate proposals +=================== + +Raising something other than RuntimeError +----------------------------------------- + +Rather than the generic ``RuntimeError``, it might make sense to raise +a new exception type ``UnexpectedStopIteration``. This has the +downside of implicitly encouraging that it be caught; the correct +action is to catch the original ``StopIteration``, not the chained +exception. + + +Supplying a specific exception to raise on return +------------------------------------------------- + +Nick Coghlan suggested a means of providing a specific +``StopIteration`` instance to the generator; if any other instance of +``StopIteration`` is raised, it is an error, but if that particular +one is raised, the generator has properly completed. This subproposal +has been withdrawn in favour of better options, but is retained for +reference. + + +Making return-triggered StopIterations obvious +---------------------------------------------- + +For certain situations, a simpler and fully backward-compatible +solution may be sufficient: when a generator returns, instead of +raising ``StopIteration``, it raises a specific subclass of +``StopIteration`` (``GeneratorReturn``) which can then be detected. +If it is not that subclass, it is an escaping exception rather than a +return statement. + +The inspiration for this alternative proposal was Nick's observation +[8]_ that if an ``asyncio`` coroutine [9]_ accidentally raises +``StopIteration``, it currently terminates silently, which may present +a hard-to-debug mystery to the developer. The main proposal turns +such accidents into clearly distinguishable ``RuntimeError`` exceptions, +but if that is rejected, this alternate proposal would enable +``asyncio`` to distinguish between a ``return`` statement and an +accidentally-raised ``StopIteration`` exception. + +Of the three outcomes listed above, two change: + +* If a yield point is reached, the value, obviously, would still be + returned. +* If the frame is returned from, ``GeneratorReturn`` (rather than + ``StopIteration``) is raised. +* If an instance of ``GeneratorReturn`` would be raised, instead an + instance of ``StopIteration`` would be raised. Any other exception + bubbles up normally. + +In the third case, the ``StopIteration`` would have the ``value`` of +the original ``GeneratorReturn``, and would reference the original +exception in its ``__cause__``. If uncaught, this would clearly show +the chaining of exceptions. + +This alternative does *not* affect the discrepancy between generator +expressions and list comprehensions, but allows generator-aware code +(such as the ``contextlib`` and ``asyncio`` modules) to reliably +differentiate between the second and third outcomes listed above. + +However, once code exists that depends on this distinction between +``GeneratorReturn`` and ``StopIteration``, a generator that invokes +another generator and relies on the latter's ``StopIteration`` to +bubble out would still be potentially wrong, depending on the use made +of the distinction between the two exception types. + + +Converting the exception inside next() +-------------------------------------- + +Mark Shannon suggested [12]_ that the problem could be solved in +``next()`` rather than at the boundary of generator functions. By +having ``next()`` catch ``StopIteration`` and raise instead +``ValueError``, all unexpected ``StopIteration`` bubbling would be +prevented; however, the backward-incompatibility concerns are far +more serious than for the current proposal, as every ``next()`` call +now needs to be rewritten to guard against ``ValueError`` instead of +``StopIteration`` - not to mention that there is no way to write one +block of code which reliably works on multiple versions of Python. +(Using a dedicated exception type, perhaps subclassing ``ValueError``, +would help this; however, all code would still need to be rewritten.) + + +Sub-proposal: decorator to explicitly request current behaviour +--------------------------------------------------------------- + +Nick Coghlan suggested [13]_ that the situations where the current +behaviour is desired could be supported by means of a decorator:: + + from itertools import allow_implicit_stop + + @allow_implicit_stop + def my_generator(): + ... + yield next(it) + ... + +Which would be semantically equivalent to:: + + def my_generator(): + try: + ... + yield next(it) + ... + except StopIteration + return + +but be faster, as it could be implemented by simply permitting the +``StopIteration`` to bubble up directly. + +Single-source Python 2/3 code would also benefit in a 3.7+ world, +since libraries like six and python-future could just define their own +version of "allow_implicit_stop" that referred to the new builtin in +3.5+, and was implemented as an identity function in other versions. + +However, due to the implementation complexities required, the ongoing +compatibility issues created, the subtlety of the decorator's effect, +and the fact that it would encourage the "quick-fix" solution of just +slapping the decorator onto all generators instead of properly fixing +the code in question, this sub-proposal has been rejected. [14]_ + + +Criticism +========= + +Unofficial and apocryphal statistics suggest that this is seldom, if +ever, a problem. [5]_ Code does exist which relies on the current +behaviour (e.g. [3]_, [6]_, [7]_), and there is the concern that this +would be unnecessary code churn to achieve little or no gain. + +Steven D'Aprano started an informal survey on comp.lang.python [10]_; +at the time of writing only two responses have been received: one was +in favor of changing list comprehensions to match generator +expressions (!), the other was in favor of this PEP's main proposal. + +The existing model has been compared to the perfectly-acceptable +issues inherent to every other case where an exception has special +meaning. For instance, an unexpected ``KeyError`` inside a +``__getitem__`` method will be interpreted as failure, rather than +permitted to bubble up. However, there is a difference. Special +methods use ``return`` to indicate normality, and ``raise`` to signal +abnormality; generators ``yield`` to indicate data, and ``return`` to +signal the abnormal state. This makes explicitly raising +``StopIteration`` entirely redundant, and potentially surprising. If +other special methods had dedicated keywords to distinguish between +their return paths, they too could turn unexpected exceptions into +``RuntimeError``; the fact that they cannot should not preclude +generators from doing so. + + +References +========== + +.. [1] PEP 380 - Syntax for Delegating to a Subgenerator + (https://www.python.org/dev/peps/pep-0380) + +.. [2] Initial mailing list comment + (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html) + +.. [3] Pure Python implementation of groupby + (https://docs.python.org/3/library/itertools.html#itertools.groupby) + +.. [4] Proposal by GvR + (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html) + +.. [5] Response by Steven D'Aprano + (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) + +.. [6] Split a sequence or generator using a predicate + (http://code.activestate.com/recipes/578416-split-a-sequence-or-generator-using-a-predicate/) + +.. [7] wrap unbounded generator to restrict its output + (http://code.activestate.com/recipes/66427-wrap-unbounded-generator-to-restrict-its-output/) + +.. [8] Post from Nick Coghlan mentioning asyncio + (https://mail.python.org/pipermail/python-ideas/2014-November/029961.html) + +.. [9] Coroutines in asyncio + (https://docs.python.org/3/library/asyncio-task.html#coroutines) + +.. [10] Thread on comp.lang.python started by Steven D'Aprano + (https://mail.python.org/pipermail/python-list/2014-November/680757.html) + +.. [11] Tracker issue with Proof-of-Concept patch + (http://bugs.python.org/issue22906) + +.. [12] Post from Mark Shannon with alternate proposal + (https://mail.python.org/pipermail/python-dev/2014-November/137129.html) + +.. [13] Idea from Nick Coghlan + (https://mail.python.org/pipermail/python-dev/2014-November/137201.html) + +.. [14] Rejection by GvR + (https://mail.python.org/pipermail/python-dev/2014-November/137243.html) + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0483.txt.orig b/pep-0483.txt.orig new file mode 100644 --- /dev/null +++ b/pep-0483.txt.orig @@ -0,0 +1,328 @@ +PEP: 483 +Title: The Theory of Type Hinting +Version: $Revision$ +Last-Modified: $Date$ +Author: Guido van Rossum +Discussions-To: Python-Ideas +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 08-Jan-2015 +Post-History: +Resolution: + +The Theory of Type Hinting +========================== + +Guido van Rossum, Dec 19, 2014 (with a few more recent updates) + +This work is licensed under a `Creative Commons +Attribution-NonCommercial-ShareAlike 4.0 International +License `_. + + +Introduction +------------ + +This document lays out the theory of the new type hinting proposal for +Python 3.5. It's not quite a full proposal or specification because +there are many details that need to be worked out, but it lays out the +theory without which it is hard to discuss more detailed specifications. +We start by explaining gradual typing; then we state some conventions +and general rules; then we define the new special types (such as Union) +that can be used in annotations; and finally we define the approach to +generic types. (The latter section needs more fleshing out; sorry!) + + +Summary of gradual typing +------------------------- + +We define a new relationship, is-consistent-with, which is similar to +is-subclass-of, except it is not transitive when the new type **Any** is +involved. (Neither relationship is symmetric.) Assigning x to y is OK if +the type of x is consistent with the type of y. (Compare this to??... if +the type of x is a subclass of the type of y,? which states one of the +fundamentals of OO programming.) The is-consistent-with relationship is +defined by three rules: + +- A type t1 is consistent with a type t2 if t1 is a subclass of t2. + (But not the other way around.) +- **Any** is consistent with every type. (But **Any** is not a subclass + of every type.) +- Every type is a subclass of **Any**. (Which also makes every type + consistent with **Any**, via rule 1.) + +That's all! See Jeremy Siek's blog post `What is Gradual +Typing `_ +for a longer explanation and motivation. Note that rule 3 places **Any** +at the root of the class graph. This makes it very similar to +**object**. The difference is that **object** is not consistent with +most types (e.g. you can't use an object() instance where an int is +expected). IOW both **Any** and **object** mean??any type is allowed? +when used to annotate an argument, but only **Any** can be passed no +matter what type is expected (in essence, **Any** shuts up complaints +from the static checker). + +Here's an example showing how these rules work out in practice: + +Say we have an Employee class, and a subclass Manager: + +- class Employee: ... +- class Manager(Employee): ... + +Let's say variable e is declared with type Employee: + +- e = Employee()? # type: Employee + +Now it's okay to assign a Manager instance to e (rule 1): + +- e = Manager() + +It's not okay to assign an Employee instance to a variable declared with +type Manager: + +- m = Manager()? # type: Manager +- m = Employee()? # Fails static check + +However, suppose we have a variable whose type is **Any**: + +- a = some\_func()? # type: Any + +Now it's okay to assign a to e (rule 2): + +- e = a? # OK + +Of course it's also okay to assign e to a (rule 3), but we didn't need +the concept of consistency for that: + +- a = e? # OK + +Notational conventions +~~~~~~~~~~~~~~~~~~~~~~ + +- t1, t2 etc.? and u1, u2 etc. are types or classes. Sometimes we write + ti or tj to refer to??any of t1, t2, etc.? +- X, Y etc. are type variables (defined with Var(), see below). +- C, D etc. are classes defined with a class statement. +- x, y etc. are objects or instances. +- We use the terms? type and? class interchangeably, and we assume + type(x) is x.\_\_class\_\_. + +General rules +~~~~~~~~~~~~~ + +- Instance-ness is? derived from class-ness, e.g. x is an instance of + t1 if? type(x) is a subclass of t1. +- No types defined below (i.e. Any, Union etc.) can be instantiated. + (But non-abstract subclasses of Generic can be.) +- No types defined below can be subclassed, except for Generic and + classes derived from it. +- Where a type is expected, None can be substituted for type(None); + e.g. Union[t1, None] == Union[t1, type(None)]. + +Types +~~~~~ + +- **Any**. Every class is a subclass of Any; however, to the static + type checker it is also consistent with every class (see above). +- **Union[t1, t2, ...]**. Classes that are subclass of at least one of + t1 etc. are subclasses of this. So are unions whose components are + all subclasses of t1 etc. (Example: Union[int, str] is a subclass of + Union[int, float, str].) The order of the arguments doesn't matter. + (Example: Union[int, str] == Union[str, int].) If ti is itself a + Union the result is flattened. (Example: Union[int, Union[float, + str]] == Union[int, float, str].) If ti and tj have a subclass + relationship, the less specific type survives. (Example: + Union[Employee, Manager] == Union[Employee].)?Union[t1] returns just + t1. Union[] is illegal, so is Union[()]. Corollary: Union[..., Any, + ...] returns Any; Union[..., object, ...] returns object; to cut a + tie, Union[Any, object] == Union[object, Any] == Any. +- **Optional[t1]**. Alias for Union[t1, None], i.e. Union[t1, + type(None)]. +- **Tuple[t1, t2, ..., tn]**. A tuple whose items are instances of t1 + etc.. Example: Tuple[int, float] means a tuple of two items, the + first is an int, the second a float; e.g., (42, 3.14). Tuple[u1, u2, + ..., um] is a subclass of Tuple[t1, t2, ..., tn] if they have the + same length (n==m) and each ui is a subclass of ti. To spell the type + of the empty tuple, use Tuple[()]. There is no way to define a + variadic tuple type. (TODO: Maybe Tuple[t1, ...] with literal + ellipsis?) +- **Callable[[t1, t2, ..., tn], tr]**. A function with positional + argument types t1 etc., and return type tr. The argument list may be + empty (n==0). There is no way to indicate optional or keyword + arguments, nor varargs (we don't need to spell those often enough to + complicate the syntax ? however, Reticulated Python has a useful idea + here). This is covariant in the return type, but contravariant in the + arguments. ?Covariant? here means that for two callable types that + differ only in the return type, the subclass relationship for the + callable types follows that of the return types. (Example: + Callable[[], Manager] is a subclass of Callable[[], Employee].) + ?Contravariant? here means that for two callable types that differ + only in the type of one argument, the subclass relationship for the + callable types goes in the opposite direction as for the argument + types. (Example: Callable[[Employee], None] is a subclass of + Callable[[Mananger], None]. Yes, you read that right.) + +We might add: + +- **Intersection[t1, t2, ...]**. Classes that are subclass of *each* of + t1, etc are subclasses of this. (Compare to Union, which has *at + least one* instead of *each* in its definition.) The order of the + arguments doesn't matter. Nested intersections are flattened, e.g. + Intersection[int, Intersection[float, str]] == Intersection[int, + float, str]. An intersection of fewer types is a subclass of an + intersection of more types, e.g. Intersection[int, str] is a subclass + of Intersection[int, float, str]. An intersection of one argument is + just that argument, e.g. Intersection[int] is int. When argument have + a subclass relationship, the more specific class survives, e.g. + Intersection[str, Employee, Manager] is Intersection[str, Manager]. + Intersection[] is illegal, so is Intersection[()]. Corollary: Any + disappears from the argument list, e.g. Intersection[int, str, Any] + == Intersection[int, str].Intersection[Any, object] is object. The + interaction between Intersection and Union is complex but should be + no surprise if you understand the interaction between intersections + and unions in set theory (note that sets of types can be infinite in + size, since there is no limit on the number of new subclasses). + +Pragmatics +~~~~~~~~~~ + +Some things are irrelevant to the theory but make practical use more +convenient. (This is not a full list; I probably missed a few and some +are still controversial or not fully specified.) + +Type aliases, e.g. + +- point = Tuple[float, float] +- def distance(p: point) -> float: ...? + +Forward references via strings, e.g. + +class C: + +- def compare(self, other: ?C?) -> int: ... + +If a default of None is specified, the type is implicitly optional, e.g. + +- def get(key: KT, default: VT = None) -> VT: ... + +Don't use dynamic type expressions; use builtins and imported types +only. No 'if'. + +- def display(message: str if WINDOWS else bytes):? # NOT OK + +Type declaration in comments, e.g. + +- x = []? # type: Sequence[int] + +Type declarations using Undefined, e.g. + +- x = Undefined(str) + +Other things, e.g. casts, overloading and stub modules; best left to an +actual PEP. + +Generic types +~~~~~~~~~~~~~ + +(TODO: Explain more. See also the `mypy docs on +generics `_.) + +**X = Var('X')**. Declares a unique type variable. The name must match +the variable name. + +**Y = Var('Y', t1, t2, ...).** Ditto, constrained to t1 etc. Behaves +?like Union[t1, t2, ...] for most purposes, but when used as a type +variable, subclasses of t1 etc. are replaced by the most-derived base +class among t1 etc. + +Example of constrained type variables: + +AnyStr = Var('AnyStr', str, bytes) + +def longest(a: AnyStr, b: AnyStr) -> AnyStr: + +- return a if len(a) >= len(b) else b + +x = longest('a', 'abc')? # The inferred type for x is str + +y = longest('a', b'abc')? # Fails static type check + +In this example, both arguments to longest() must have the same type +(str or bytes), and moreover, even if the arguments are instances of a +common str subclass, the return type is still str, not that subclass +(see next example). + +For comparison, if the type variable was unconstrained, the common +subclass would be chosen as the return type, e.g.: + +S = Var('S') + +def longest(a: S, b: S) -> S: + +- return a if len(a) >= b else b + +class MyStr(str): ... + +x = longest(MyStr('a'), MyStr('abc')) + +The inferred type of x is MyStr (whereas in the AnyStr example it would +be str). + +Also for comparison, if a Union is used, the return type also has to be +a Union: + +U = Union[str, bytes] + +def longest(a: U, b: U) -> U: + +- return a if len(a) >- b else b + +x = longest('a', 'abc') + +The inferred type of x is still Union[str, bytes], even though both +arguments are str. + +**class C(Generic[X, Y, ...]):** ... Define a generic class C over type +variables X etc. C itself becomes parameterizable, e.g. C[int, str, ...] +is a specific class with substitutions X?int etc. + +TODO: Explain use of generic types in function signatures. E.g. +Sequence[X], Sequence[int], Sequence[Tuple[X, Y, Z]], and mixtures. +Think about co\*variance. No gimmicks like deriving from +Sequence[Union[int, str]] or Sequence[Union[int, X]]. + +**Protocol**. Similar to Generic but uses structural equivalence. (TODO: +Explain, and think about co\*variance.) + +Predefined generic types and Protocols in typing.py +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +(See also the `mypy typing.py +module `_.) + +- Everything from collections.abc (but Set renamed to AbstractSet). +- Dict, List, Set, a few more. (FrozenSet?) +- Pattern, Match. (Why?) +- IO, TextIO, BinaryIO. (Why?) + +Another reference +~~~~~~~~~~~~~~~~~ + +Lest mypy gets all the attention, I should mention?\ `Reticulated +Python `_ by Michael Vitousek +as an example of a slightly different approach to gradual typing for +Python. It is described in an actual `academic +paper `_ +written by Vitousek with Jeremy Siek and Jim Baker (the latter of Jython +fame). + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Mon Jan 19 23:32:08 2015 From: python-checkins at python.org (chris.angelico) Date: Mon, 19 Jan 2015 22:32:08 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Oops_-_remove_accidentally-co?= =?utf-8?q?mmitted_files?= Message-ID: <20150119223141.103311.28358@psf.io> https://hg.python.org/peps/rev/5fc86201497b changeset: 5676:5fc86201497b user: Chris Angelico date: Tue Jan 20 09:31:33 2015 +1100 summary: Oops - remove accidentally-committed files files: pep-0479.txt.orig | 603 ---------------------------------- pep-0483.txt.orig | 328 ------------------ 2 files changed, 0 insertions(+), 931 deletions(-) diff --git a/pep-0479.txt.orig b/pep-0479.txt.orig deleted file mode 100644 --- a/pep-0479.txt.orig +++ /dev/null @@ -1,603 +0,0 @@ -PEP: 479 -Title: Change StopIteration handling inside generators -Version: $Revision$ -Last-Modified: $Date$ -Author: Chris Angelico , Guido van Rossum -Status: Accepted -Type: Standards Track -Content-Type: text/x-rst -Created: 15-Nov-2014 -Python-Version: 3.5 -Post-History: 15-Nov-2014, 19-Nov-2014, 5-Dec-2014 - - -Abstract -======== - -This PEP proposes a change to generators: when ``StopIteration`` is -raised inside a generator, it is replaced it with ``RuntimeError``. -(More precisely, this happens when the exception is about to bubble -out of the generator's stack frame.) Because the change is backwards -incompatible, the feature is initially introduced using a -``__future__`` statement. - - -Acceptance -========== - -This PEP was accepted by the BDFL on November 22. Because of the -exceptionally short period from first draft to acceptance, the main -objections brought up after acceptance were carefully considered and -have been reflected in the "Alternate proposals" section below. -However, none of the discussion changed the BDFL's mind and the PEP's -acceptance is now final. (Suggestions for clarifying edits are still -welcome -- unlike IETF RFCs, the text of a PEP is not cast in stone -after its acceptance, although the core design/plan/specification -should not change after acceptance.) - - -Rationale -========= - -The interaction of generators and ``StopIteration`` is currently -somewhat surprising, and can conceal obscure bugs. An unexpected -exception should not result in subtly altered behaviour, but should -cause a noisy and easily-debugged traceback. Currently, -``StopIteration`` can be absorbed by the generator construct. - -The main goal of the proposal is to ease debugging in the situation -where an unguarded ``next()`` call (perhaps several stack frames deep) -raises ``StopIteration`` and causes the iteration controlled by the -generator to terminate silently. (When another exception is raised, a -traceback is printed pinpointing the cause of the problem.) - -This is particularly pernicious in combination with the ``yield from`` -construct of PEP 380 [1]_, as it breaks the abstraction that a -subgenerator may be factored out of a generator. That PEP notes this -limitation, but notes that "use cases for these [are] rare to non- -existent". Unfortunately while intentional use is rare, it is easy to -stumble on these cases by accident:: - - import contextlib - - @contextlib.contextmanager - def transaction(): - print('begin') - try: - yield from do_it() - except: - print('rollback') - raise - else: - print('commit') - - def do_it(): - print('Refactored initial setup') - yield # Body of with-statement is executed here - print('Refactored finalization of successful transaction') - - - import pathlib - - with transaction(): - print('commit file {}'.format( - # I can never remember what the README extension is - next(pathlib.Path('/some/dir').glob('README*')))) - -Here factoring out ``do_it`` into a subgenerator has introduced a subtle -bug: if a bug in the wrapped block allows ``StopIteration`` to escape -(here because the README doesn't exist), under the current behavior -``do_it`` will properly abort, but the exception will be swallowed by -the ``yield from`` and the original context manager will commit the -unfinished transaction! Similarly problematic behavior occurs when an -``asyncio`` coroutine raises ``StopIteration``, causing it to terminate -silently. In both cases, the refactoring abstraction of ``yield from`` -breaks in the presence of bugs in client code. - -Additionally, the proposal reduces the difference between list -comprehensions and generator expressions, preventing surprises such as -the one that started this discussion [2]_. Henceforth, the following -statements will produce the same result if either produces a result at -all:: - - a = list(F(x) for x in xs if P(x)) - a = [F(x) for x in xs if P(x)] - -With the current state of affairs, it is possible to write a function -``F(x)`` or a predicate ``P(x)`` that causes the first form to produce -a (truncated) result, while the second form raises an exception -(namely, ``StopIteration``). With the proposed change, both forms -will raise an exception at this point (albeit ``RuntimeError`` in the -first case and ``StopIteration`` in the second). - -Finally, the proposal also clears up the confusion about how to -terminate a generator: the proper way is ``return``, not -``raise StopIteration``. - -As an added bonus, the above changes bring generator functions much -more in line with regular functions. If you wish to take a piece of -code presented as a generator and turn it into something else, you -can usually do this fairly simply, by replacing every ``yield`` with -a call to ``print()`` or ``list.append()``; however, if there are any -bare ``next()`` calls in the code, you have to be aware of them. If -the code was originally written without relying on ``StopIteration`` -terminating the function, the transformation would be that much -easier. - - -Background information -====================== - -When a generator frame is (re)started as a result of a ``__next__()`` -(or ``send()`` or ``throw()``) call, one of three outcomes can occur: - -* A yield point is reached, and the yielded value is returned. -* The frame is returned from; ``StopIteration`` is raised. -* An exception is raised, which bubbles out. - -In the latter two cases the frame is abandoned (and the generator -object's ``gi_frame`` attribute is set to None). - - -Proposal -======== - -If a ``StopIteration`` is about to bubble out of a generator frame, it -is replaced with ``RuntimeError``, which causes the ``next()`` call -(which invoked the generator) to fail, passing that exception out. -From then on it's just like any old exception. [4]_ - -This affects the third outcome listed above, without altering any -other effects. Furthermore, it only affects this outcome when the -exception raised is ``StopIteration`` (or a subclass thereof). - -Note that the proposed replacement happens at the point where the -exception is about to bubble out of the frame, i.e. after any -``except`` or ``finally`` blocks that could affect it have been -exited. The ``StopIteration`` raised by returning from the frame is -not affected (the point being that ``StopIteration`` means that the -generator terminated "normally", i.e. it did not raise an exception). - -A subtle issue is what will happen if the caller, having caught the -``RuntimeError``, calls the generator object's ``__next__()`` method -again. The answer is that from this point on it will raise -``StopIteration`` -- the behavior is the same as when any other -exception was raised by the generator. - -Another logical consequence of the proposal: if someone uses -``g.throw(StopIteration)`` to throw a ``StopIteration`` exception into -a generator, if the generator doesn't catch it (which it could do -using a ``try/except`` around the ``yield``), it will be transformed -into ``RuntimeError``. - -During the transition phase, the new feature must be enabled -per-module using:: - - from __future__ import generator_stop - -Any generator function constructed under the influence of this -directive will have the ``REPLACE_STOPITERATION`` flag set on its code -object, and generators with the flag set will behave according to this -proposal. Once the feature becomes standard, the flag may be dropped; -code should not inspect generators for it. - - -Consequences for existing code -============================== - -This change will affect existing code that depends on -``StopIteration`` bubbling up. The pure Python reference -implementation of ``groupby`` [3]_ currently has comments "Exit on -``StopIteration``" where it is expected that the exception will -propagate and then be handled. This will be unusual, but not unknown, -and such constructs will fail. Other examples abound, e.g. [6]_, [7]_. - -(Nick Coghlan comments: """If you wanted to factor out a helper -function that terminated the generator you'd have to do "return -yield from helper()" rather than just "helper()".""") - -There are also examples of generator expressions floating around that -rely on a ``StopIteration`` raised by the expression, the target or the -predicate (rather than by the ``__next__()`` call implied in the ``for`` -loop proper). - -Writing backwards and forwards compatible code ----------------------------------------------- - -With the exception of hacks that raise ``StopIteration`` to exit a -generator expression, it is easy to write code that works equally well -under older Python versions as under the new semantics. - -This is done by enclosing those places in the generator body where a -``StopIteration`` is expected (e.g. bare ``next()`` calls or in some -cases helper functions that are expected to raise ``StopIteration``) -in a ``try/except`` construct that returns when ``StopIteration`` is -raised. The ``try/except`` construct should appear directly in the -generator function; doing this in a helper function that is not itself -a generator does not work. If ``raise StopIteration`` occurs directly -in a generator, simply replace it with ``return``. - - -Examples of breakage --------------------- - -Generators which explicitly raise ``StopIteration`` can generally be -changed to simply return instead. This will be compatible with all -existing Python versions, and will not be affected by ``__future__``. -Here are some illustrations from the standard library. - -Lib/ipaddress.py:: - - if other == self: - raise StopIteration - -Becomes:: - - if other == self: - return - -In some cases, this can be combined with ``yield from`` to simplify -the code, such as Lib/difflib.py:: - - if context is None: - while True: - yield next(line_pair_iterator) - -Becomes:: - - if context is None: - yield from line_pair_iterator - return - -(The ``return`` is necessary for a strictly-equivalent translation, -though in this particular file, there is no further code, and the -``return`` can be omitted.) For compatibility with pre-3.3 versions -of Python, this could be written with an explicit ``for`` loop:: - - if context is None: - for line in line_pair_iterator: - yield line - return - -More complicated iteration patterns will need explicit ``try/except`` -constructs. For example, a hypothetical parser like this:: - - def parser(f): - while True: - data = next(f) - while True: - line = next(f) - if line == "- end -": break - data += line - yield data - -would need to be rewritten as:: - - def parser(f): - while True: - try: - data = next(f) - while True: - line = next(f) - if line == "- end -": break - data += line - yield data - except StopIteration: - return - -or possibly:: - - def parser(f): - for data in f: - while True: - line = next(f) - if line == "- end -": break - data += line - yield data - -The latter form obscures the iteration by purporting to iterate over -the file with a ``for`` loop, but then also fetches more data from -the same iterator during the loop body. It does, however, clearly -differentiate between a "normal" termination (``StopIteration`` -instead of the initial line) and an "abnormal" termination (failing -to find the end marker in the inner loop, which will now raise -``RuntimeError``). - -This effect of ``StopIteration`` has been used to cut a generator -expression short, creating a form of ``takewhile``:: - - def stop(): - raise StopIteration - print(list(x for x in range(10) if x < 5 or stop())) - # prints [0, 1, 2, 3, 4] - -Under the current proposal, this form of non-local flow control is -not supported, and would have to be rewritten in statement form:: - - def gen(): - for x in range(10): - if x >= 5: return - yield x - print(list(gen())) - # prints [0, 1, 2, 3, 4] - -While this is a small loss of functionality, it is functionality that -often comes at the cost of readability, and just as ``lambda`` has -restrictions compared to ``def``, so does a generator expression have -restrictions compared to a generator function. In many cases, the -transformation to full generator function will be trivially easy, and -may improve structural clarity. - - -Explanation of generators, iterators, and StopIteration -======================================================= - -Under this proposal, generators and iterators would be distinct, but -related, concepts. Like the mixing of text and bytes in Python 2, -the mixing of generators and iterators has resulted in certain -perceived conveniences, but proper separation will make bugs more -visible. - -An iterator is an object with a ``__next__`` method. Like many other -special methods, it may either return a value, or raise a specific -exception - in this case, ``StopIteration`` - to signal that it has -no value to return. In this, it is similar to ``__getattr__`` (can -raise ``AttributeError``), ``__getitem__`` (can raise ``KeyError``), -and so on. A helper function for an iterator can be written to -follow the same protocol; for example:: - - def helper(x, y): - if x > y: return 1 / (x - y) - raise StopIteration - - def __next__(self): - if self.a: return helper(self.b, self.c) - return helper(self.d, self.e) - -Both forms of signalling are carried through: a returned value is -returned, an exception bubbles up. The helper is written to match -the protocol of the calling function. - -A generator function is one which contains a ``yield`` expression. -Each time it is (re)started, it may either yield a value, or return -(including "falling off the end"). A helper function for a generator -can also be written, but it must also follow generator protocol:: - - def helper(x, y): - if x > y: yield 1 / (x - y) - - def gen(self): - if self.a: return (yield from helper(self.b, self.c)) - return (yield from helper(self.d, self.e)) - -In both cases, any unexpected exception will bubble up. Due to the -nature of generators and iterators, an unexpected ``StopIteration`` -inside a generator will be converted into ``RuntimeError``, but -beyond that, all exceptions will propagate normally. - - -Transition plan -=============== - -- Python 3.5: Enable new semantics under ``__future__`` import; silent - deprecation warning if ``StopIteration`` bubbles out of a generator - not under ``__future__`` import. - -- Python 3.6: Non-silent deprecation warning. - -- Python 3.7: Enable new semantics everywhere. - - -Alternate proposals -=================== - -Raising something other than RuntimeError ------------------------------------------ - -Rather than the generic ``RuntimeError``, it might make sense to raise -a new exception type ``UnexpectedStopIteration``. This has the -downside of implicitly encouraging that it be caught; the correct -action is to catch the original ``StopIteration``, not the chained -exception. - - -Supplying a specific exception to raise on return -------------------------------------------------- - -Nick Coghlan suggested a means of providing a specific -``StopIteration`` instance to the generator; if any other instance of -``StopIteration`` is raised, it is an error, but if that particular -one is raised, the generator has properly completed. This subproposal -has been withdrawn in favour of better options, but is retained for -reference. - - -Making return-triggered StopIterations obvious ----------------------------------------------- - -For certain situations, a simpler and fully backward-compatible -solution may be sufficient: when a generator returns, instead of -raising ``StopIteration``, it raises a specific subclass of -``StopIteration`` (``GeneratorReturn``) which can then be detected. -If it is not that subclass, it is an escaping exception rather than a -return statement. - -The inspiration for this alternative proposal was Nick's observation -[8]_ that if an ``asyncio`` coroutine [9]_ accidentally raises -``StopIteration``, it currently terminates silently, which may present -a hard-to-debug mystery to the developer. The main proposal turns -such accidents into clearly distinguishable ``RuntimeError`` exceptions, -but if that is rejected, this alternate proposal would enable -``asyncio`` to distinguish between a ``return`` statement and an -accidentally-raised ``StopIteration`` exception. - -Of the three outcomes listed above, two change: - -* If a yield point is reached, the value, obviously, would still be - returned. -* If the frame is returned from, ``GeneratorReturn`` (rather than - ``StopIteration``) is raised. -* If an instance of ``GeneratorReturn`` would be raised, instead an - instance of ``StopIteration`` would be raised. Any other exception - bubbles up normally. - -In the third case, the ``StopIteration`` would have the ``value`` of -the original ``GeneratorReturn``, and would reference the original -exception in its ``__cause__``. If uncaught, this would clearly show -the chaining of exceptions. - -This alternative does *not* affect the discrepancy between generator -expressions and list comprehensions, but allows generator-aware code -(such as the ``contextlib`` and ``asyncio`` modules) to reliably -differentiate between the second and third outcomes listed above. - -However, once code exists that depends on this distinction between -``GeneratorReturn`` and ``StopIteration``, a generator that invokes -another generator and relies on the latter's ``StopIteration`` to -bubble out would still be potentially wrong, depending on the use made -of the distinction between the two exception types. - - -Converting the exception inside next() --------------------------------------- - -Mark Shannon suggested [12]_ that the problem could be solved in -``next()`` rather than at the boundary of generator functions. By -having ``next()`` catch ``StopIteration`` and raise instead -``ValueError``, all unexpected ``StopIteration`` bubbling would be -prevented; however, the backward-incompatibility concerns are far -more serious than for the current proposal, as every ``next()`` call -now needs to be rewritten to guard against ``ValueError`` instead of -``StopIteration`` - not to mention that there is no way to write one -block of code which reliably works on multiple versions of Python. -(Using a dedicated exception type, perhaps subclassing ``ValueError``, -would help this; however, all code would still need to be rewritten.) - - -Sub-proposal: decorator to explicitly request current behaviour ---------------------------------------------------------------- - -Nick Coghlan suggested [13]_ that the situations where the current -behaviour is desired could be supported by means of a decorator:: - - from itertools import allow_implicit_stop - - @allow_implicit_stop - def my_generator(): - ... - yield next(it) - ... - -Which would be semantically equivalent to:: - - def my_generator(): - try: - ... - yield next(it) - ... - except StopIteration - return - -but be faster, as it could be implemented by simply permitting the -``StopIteration`` to bubble up directly. - -Single-source Python 2/3 code would also benefit in a 3.7+ world, -since libraries like six and python-future could just define their own -version of "allow_implicit_stop" that referred to the new builtin in -3.5+, and was implemented as an identity function in other versions. - -However, due to the implementation complexities required, the ongoing -compatibility issues created, the subtlety of the decorator's effect, -and the fact that it would encourage the "quick-fix" solution of just -slapping the decorator onto all generators instead of properly fixing -the code in question, this sub-proposal has been rejected. [14]_ - - -Criticism -========= - -Unofficial and apocryphal statistics suggest that this is seldom, if -ever, a problem. [5]_ Code does exist which relies on the current -behaviour (e.g. [3]_, [6]_, [7]_), and there is the concern that this -would be unnecessary code churn to achieve little or no gain. - -Steven D'Aprano started an informal survey on comp.lang.python [10]_; -at the time of writing only two responses have been received: one was -in favor of changing list comprehensions to match generator -expressions (!), the other was in favor of this PEP's main proposal. - -The existing model has been compared to the perfectly-acceptable -issues inherent to every other case where an exception has special -meaning. For instance, an unexpected ``KeyError`` inside a -``__getitem__`` method will be interpreted as failure, rather than -permitted to bubble up. However, there is a difference. Special -methods use ``return`` to indicate normality, and ``raise`` to signal -abnormality; generators ``yield`` to indicate data, and ``return`` to -signal the abnormal state. This makes explicitly raising -``StopIteration`` entirely redundant, and potentially surprising. If -other special methods had dedicated keywords to distinguish between -their return paths, they too could turn unexpected exceptions into -``RuntimeError``; the fact that they cannot should not preclude -generators from doing so. - - -References -========== - -.. [1] PEP 380 - Syntax for Delegating to a Subgenerator - (https://www.python.org/dev/peps/pep-0380) - -.. [2] Initial mailing list comment - (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html) - -.. [3] Pure Python implementation of groupby - (https://docs.python.org/3/library/itertools.html#itertools.groupby) - -.. [4] Proposal by GvR - (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html) - -.. [5] Response by Steven D'Aprano - (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) - -.. [6] Split a sequence or generator using a predicate - (http://code.activestate.com/recipes/578416-split-a-sequence-or-generator-using-a-predicate/) - -.. [7] wrap unbounded generator to restrict its output - (http://code.activestate.com/recipes/66427-wrap-unbounded-generator-to-restrict-its-output/) - -.. [8] Post from Nick Coghlan mentioning asyncio - (https://mail.python.org/pipermail/python-ideas/2014-November/029961.html) - -.. [9] Coroutines in asyncio - (https://docs.python.org/3/library/asyncio-task.html#coroutines) - -.. [10] Thread on comp.lang.python started by Steven D'Aprano - (https://mail.python.org/pipermail/python-list/2014-November/680757.html) - -.. [11] Tracker issue with Proof-of-Concept patch - (http://bugs.python.org/issue22906) - -.. [12] Post from Mark Shannon with alternate proposal - (https://mail.python.org/pipermail/python-dev/2014-November/137129.html) - -.. [13] Idea from Nick Coghlan - (https://mail.python.org/pipermail/python-dev/2014-November/137201.html) - -.. [14] Rejection by GvR - (https://mail.python.org/pipermail/python-dev/2014-November/137243.html) - -Copyright -========= - -This document has been placed in the public domain. - - - -.. - Local Variables: - mode: indented-text - indent-tabs-mode: nil - sentence-end-double-space: t - fill-column: 70 - coding: utf-8 - End: diff --git a/pep-0483.txt.orig b/pep-0483.txt.orig deleted file mode 100644 --- a/pep-0483.txt.orig +++ /dev/null @@ -1,328 +0,0 @@ -PEP: 483 -Title: The Theory of Type Hinting -Version: $Revision$ -Last-Modified: $Date$ -Author: Guido van Rossum -Discussions-To: Python-Ideas -Status: Draft -Type: Informational -Content-Type: text/x-rst -Created: 08-Jan-2015 -Post-History: -Resolution: - -The Theory of Type Hinting -========================== - -Guido van Rossum, Dec 19, 2014 (with a few more recent updates) - -This work is licensed under a `Creative Commons -Attribution-NonCommercial-ShareAlike 4.0 International -License `_. - - -Introduction ------------- - -This document lays out the theory of the new type hinting proposal for -Python 3.5. It's not quite a full proposal or specification because -there are many details that need to be worked out, but it lays out the -theory without which it is hard to discuss more detailed specifications. -We start by explaining gradual typing; then we state some conventions -and general rules; then we define the new special types (such as Union) -that can be used in annotations; and finally we define the approach to -generic types. (The latter section needs more fleshing out; sorry!) - - -Summary of gradual typing -------------------------- - -We define a new relationship, is-consistent-with, which is similar to -is-subclass-of, except it is not transitive when the new type **Any** is -involved. (Neither relationship is symmetric.) Assigning x to y is OK if -the type of x is consistent with the type of y. (Compare this to??... if -the type of x is a subclass of the type of y,? which states one of the -fundamentals of OO programming.) The is-consistent-with relationship is -defined by three rules: - -- A type t1 is consistent with a type t2 if t1 is a subclass of t2. - (But not the other way around.) -- **Any** is consistent with every type. (But **Any** is not a subclass - of every type.) -- Every type is a subclass of **Any**. (Which also makes every type - consistent with **Any**, via rule 1.) - -That's all! See Jeremy Siek's blog post `What is Gradual -Typing `_ -for a longer explanation and motivation. Note that rule 3 places **Any** -at the root of the class graph. This makes it very similar to -**object**. The difference is that **object** is not consistent with -most types (e.g. you can't use an object() instance where an int is -expected). IOW both **Any** and **object** mean??any type is allowed? -when used to annotate an argument, but only **Any** can be passed no -matter what type is expected (in essence, **Any** shuts up complaints -from the static checker). - -Here's an example showing how these rules work out in practice: - -Say we have an Employee class, and a subclass Manager: - -- class Employee: ... -- class Manager(Employee): ... - -Let's say variable e is declared with type Employee: - -- e = Employee()? # type: Employee - -Now it's okay to assign a Manager instance to e (rule 1): - -- e = Manager() - -It's not okay to assign an Employee instance to a variable declared with -type Manager: - -- m = Manager()? # type: Manager -- m = Employee()? # Fails static check - -However, suppose we have a variable whose type is **Any**: - -- a = some\_func()? # type: Any - -Now it's okay to assign a to e (rule 2): - -- e = a? # OK - -Of course it's also okay to assign e to a (rule 3), but we didn't need -the concept of consistency for that: - -- a = e? # OK - -Notational conventions -~~~~~~~~~~~~~~~~~~~~~~ - -- t1, t2 etc.? and u1, u2 etc. are types or classes. Sometimes we write - ti or tj to refer to??any of t1, t2, etc.? -- X, Y etc. are type variables (defined with Var(), see below). -- C, D etc. are classes defined with a class statement. -- x, y etc. are objects or instances. -- We use the terms? type and? class interchangeably, and we assume - type(x) is x.\_\_class\_\_. - -General rules -~~~~~~~~~~~~~ - -- Instance-ness is? derived from class-ness, e.g. x is an instance of - t1 if? type(x) is a subclass of t1. -- No types defined below (i.e. Any, Union etc.) can be instantiated. - (But non-abstract subclasses of Generic can be.) -- No types defined below can be subclassed, except for Generic and - classes derived from it. -- Where a type is expected, None can be substituted for type(None); - e.g. Union[t1, None] == Union[t1, type(None)]. - -Types -~~~~~ - -- **Any**. Every class is a subclass of Any; however, to the static - type checker it is also consistent with every class (see above). -- **Union[t1, t2, ...]**. Classes that are subclass of at least one of - t1 etc. are subclasses of this. So are unions whose components are - all subclasses of t1 etc. (Example: Union[int, str] is a subclass of - Union[int, float, str].) The order of the arguments doesn't matter. - (Example: Union[int, str] == Union[str, int].) If ti is itself a - Union the result is flattened. (Example: Union[int, Union[float, - str]] == Union[int, float, str].) If ti and tj have a subclass - relationship, the less specific type survives. (Example: - Union[Employee, Manager] == Union[Employee].)?Union[t1] returns just - t1. Union[] is illegal, so is Union[()]. Corollary: Union[..., Any, - ...] returns Any; Union[..., object, ...] returns object; to cut a - tie, Union[Any, object] == Union[object, Any] == Any. -- **Optional[t1]**. Alias for Union[t1, None], i.e. Union[t1, - type(None)]. -- **Tuple[t1, t2, ..., tn]**. A tuple whose items are instances of t1 - etc.. Example: Tuple[int, float] means a tuple of two items, the - first is an int, the second a float; e.g., (42, 3.14). Tuple[u1, u2, - ..., um] is a subclass of Tuple[t1, t2, ..., tn] if they have the - same length (n==m) and each ui is a subclass of ti. To spell the type - of the empty tuple, use Tuple[()]. There is no way to define a - variadic tuple type. (TODO: Maybe Tuple[t1, ...] with literal - ellipsis?) -- **Callable[[t1, t2, ..., tn], tr]**. A function with positional - argument types t1 etc., and return type tr. The argument list may be - empty (n==0). There is no way to indicate optional or keyword - arguments, nor varargs (we don't need to spell those often enough to - complicate the syntax ? however, Reticulated Python has a useful idea - here). This is covariant in the return type, but contravariant in the - arguments. ?Covariant? here means that for two callable types that - differ only in the return type, the subclass relationship for the - callable types follows that of the return types. (Example: - Callable[[], Manager] is a subclass of Callable[[], Employee].) - ?Contravariant? here means that for two callable types that differ - only in the type of one argument, the subclass relationship for the - callable types goes in the opposite direction as for the argument - types. (Example: Callable[[Employee], None] is a subclass of - Callable[[Mananger], None]. Yes, you read that right.) - -We might add: - -- **Intersection[t1, t2, ...]**. Classes that are subclass of *each* of - t1, etc are subclasses of this. (Compare to Union, which has *at - least one* instead of *each* in its definition.) The order of the - arguments doesn't matter. Nested intersections are flattened, e.g. - Intersection[int, Intersection[float, str]] == Intersection[int, - float, str]. An intersection of fewer types is a subclass of an - intersection of more types, e.g. Intersection[int, str] is a subclass - of Intersection[int, float, str]. An intersection of one argument is - just that argument, e.g. Intersection[int] is int. When argument have - a subclass relationship, the more specific class survives, e.g. - Intersection[str, Employee, Manager] is Intersection[str, Manager]. - Intersection[] is illegal, so is Intersection[()]. Corollary: Any - disappears from the argument list, e.g. Intersection[int, str, Any] - == Intersection[int, str].Intersection[Any, object] is object. The - interaction between Intersection and Union is complex but should be - no surprise if you understand the interaction between intersections - and unions in set theory (note that sets of types can be infinite in - size, since there is no limit on the number of new subclasses). - -Pragmatics -~~~~~~~~~~ - -Some things are irrelevant to the theory but make practical use more -convenient. (This is not a full list; I probably missed a few and some -are still controversial or not fully specified.) - -Type aliases, e.g. - -- point = Tuple[float, float] -- def distance(p: point) -> float: ...? - -Forward references via strings, e.g. - -class C: - -- def compare(self, other: ?C?) -> int: ... - -If a default of None is specified, the type is implicitly optional, e.g. - -- def get(key: KT, default: VT = None) -> VT: ... - -Don't use dynamic type expressions; use builtins and imported types -only. No 'if'. - -- def display(message: str if WINDOWS else bytes):? # NOT OK - -Type declaration in comments, e.g. - -- x = []? # type: Sequence[int] - -Type declarations using Undefined, e.g. - -- x = Undefined(str) - -Other things, e.g. casts, overloading and stub modules; best left to an -actual PEP. - -Generic types -~~~~~~~~~~~~~ - -(TODO: Explain more. See also the `mypy docs on -generics `_.) - -**X = Var('X')**. Declares a unique type variable. The name must match -the variable name. - -**Y = Var('Y', t1, t2, ...).** Ditto, constrained to t1 etc. Behaves -?like Union[t1, t2, ...] for most purposes, but when used as a type -variable, subclasses of t1 etc. are replaced by the most-derived base -class among t1 etc. - -Example of constrained type variables: - -AnyStr = Var('AnyStr', str, bytes) - -def longest(a: AnyStr, b: AnyStr) -> AnyStr: - -- return a if len(a) >= len(b) else b - -x = longest('a', 'abc')? # The inferred type for x is str - -y = longest('a', b'abc')? # Fails static type check - -In this example, both arguments to longest() must have the same type -(str or bytes), and moreover, even if the arguments are instances of a -common str subclass, the return type is still str, not that subclass -(see next example). - -For comparison, if the type variable was unconstrained, the common -subclass would be chosen as the return type, e.g.: - -S = Var('S') - -def longest(a: S, b: S) -> S: - -- return a if len(a) >= b else b - -class MyStr(str): ... - -x = longest(MyStr('a'), MyStr('abc')) - -The inferred type of x is MyStr (whereas in the AnyStr example it would -be str). - -Also for comparison, if a Union is used, the return type also has to be -a Union: - -U = Union[str, bytes] - -def longest(a: U, b: U) -> U: - -- return a if len(a) >- b else b - -x = longest('a', 'abc') - -The inferred type of x is still Union[str, bytes], even though both -arguments are str. - -**class C(Generic[X, Y, ...]):** ... Define a generic class C over type -variables X etc. C itself becomes parameterizable, e.g. C[int, str, ...] -is a specific class with substitutions X?int etc. - -TODO: Explain use of generic types in function signatures. E.g. -Sequence[X], Sequence[int], Sequence[Tuple[X, Y, Z]], and mixtures. -Think about co\*variance. No gimmicks like deriving from -Sequence[Union[int, str]] or Sequence[Union[int, X]]. - -**Protocol**. Similar to Generic but uses structural equivalence. (TODO: -Explain, and think about co\*variance.) - -Predefined generic types and Protocols in typing.py -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -(See also the `mypy typing.py -module `_.) - -- Everything from collections.abc (but Set renamed to AbstractSet). -- Dict, List, Set, a few more. (FrozenSet?) -- Pattern, Match. (Why?) -- IO, TextIO, BinaryIO. (Why?) - -Another reference -~~~~~~~~~~~~~~~~~ - -Lest mypy gets all the attention, I should mention?\ `Reticulated -Python `_ by Michael Vitousek -as an example of a slightly different approach to gradual typing for -Python. It is described in an actual `academic -paper `_ -written by Vitousek with Jeremy Siek and Jim Baker (the latter of Jython -fame). - - -.. - Local Variables: - mode: indented-text - indent-tabs-mode: nil - sentence-end-double-space: t - fill-column: 70 - coding: utf-8 - End: -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Tue Jan 20 05:30:40 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 20 Jan 2015 04:30:40 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320898=3A_Add_a_?= =?utf-8?q?=22HTTP_status_codes=22_section_to_avoid_duplication_in_HTTP?= Message-ID: <20150120043038.118090.22816@psf.io> https://hg.python.org/cpython/rev/c8647dab4780 changeset: 94219:c8647dab4780 user: Berker Peksag date: Tue Jan 20 06:30:46 2015 +0200 summary: Issue #20898: Add a "HTTP status codes" section to avoid duplication in HTTP docs. This commit also removes a couple of non-standard status codes. They were added as part of edf669b13482, so there is no backwards compatibility issue. Patch by Demian Brecht. files: Doc/library/http.client.rst | 212 +----------------------- Doc/library/http.rst | 158 ++++++++--------- Lib/http/__init__.py | 30 --- 3 files changed, 79 insertions(+), 321 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -180,221 +180,15 @@ The default port for the HTTPS protocol (always ``443``). -and also the following constants for integer status codes: - -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| Constant | Value | Definition | -+==========================================+=========+=======================================================================+ -| :const:`CONTINUE` | ``100`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.1 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SWITCHING_PROTOCOLS` | ``101`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.2 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROCESSING` | ``102`` | WEBDAV, `RFC 2518, Section 10.1 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`OK` | ``200`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.1 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CREATED` | ``201`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.2 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`ACCEPTED` | ``202`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.3 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NON_AUTHORITATIVE_INFORMATION` | ``203`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.4 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NO_CONTENT` | ``204`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.5 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`RESET_CONTENT` | ``205`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.6 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PARTIAL_CONTENT` | ``206`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.7 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTI_STATUS` | ``207`` | WEBDAV `RFC 2518, Section 10.2 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`IM_USED` | ``226`` | Delta encoding in HTTP, | -| | | :rfc:`3229`, Section 10.4.1 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTIPLE_CHOICES` | ``300`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.1 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MOVED_PERMANENTLY` | ``301`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.2 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FOUND` | ``302`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.3 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SEE_OTHER` | ``303`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.4 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_MODIFIED` | ``304`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.5 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`USE_PROXY` | ``305`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.6 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TEMPORARY_REDIRECT` | ``307`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.8 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_REQUEST` | ``400`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.1 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNAUTHORIZED` | ``401`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.2 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PAYMENT_REQUIRED` | ``402`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.3 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FORBIDDEN` | ``403`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.4 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_FOUND` | ``404`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.5 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`METHOD_NOT_ALLOWED` | ``405`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.6 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_ACCEPTABLE` | ``406`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.7 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROXY_AUTHENTICATION_REQUIRED` | ``407`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.8 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_TIMEOUT` | ``408`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.9 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CONFLICT` | ``409`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.10 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GONE` | ``410`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.11 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LENGTH_REQUIRED` | ``411`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.12 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_FAILED` | ``412`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.13 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_ENTITY_TOO_LARGE` | ``413`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.14 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_URI_TOO_LONG` | ``414`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.15 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNSUPPORTED_MEDIA_TYPE` | ``415`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.16 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUESTED_RANGE_NOT_SATISFIABLE` | ``416`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.17 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`EXPECTATION_FAILED` | ``417`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.18 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNPROCESSABLE_ENTITY` | ``422`` | WEBDAV, `RFC 2518, Section 10.3 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LOCKED` | ``423`` | WEBDAV `RFC 2518, Section 10.4 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FAILED_DEPENDENCY` | ``424`` | WEBDAV, `RFC 2518, Section 10.5 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | -| | | :rfc:`2817`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 3 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 4 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 5 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.1 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_IMPLEMENTED` | ``501`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.2 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_GATEWAY` | ``502`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.3 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SERVICE_UNAVAILABLE` | ``503`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.4 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GATEWAY_TIMEOUT` | ``504`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.5 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`HTTP_VERSION_NOT_SUPPORTED` | ``505`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.6 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INSUFFICIENT_STORAGE` | ``507`` | WEBDAV, `RFC 2518, Section 10.6 | -| | | `_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | -| | | :rfc:`2774`, Section 7 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ - -.. versionchanged:: 3.3 - Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. - - .. data:: responses This dictionary maps the HTTP 1.1 status codes to the W3C names. Example: ``http.client.responses[http.client.NOT_FOUND]`` is ``'Not Found'``. +See :ref:`http-status-codes` for a list of HTTP status codes that are +available in this module as constants. + .. _httpconnection-objects: diff --git a/Doc/library/http.rst b/Doc/library/http.rst --- a/Doc/library/http.rst +++ b/Doc/library/http.rst @@ -10,7 +10,7 @@ **Source code:** :source:`Lib/http/__init__.py` -:mod:`http` is a also package that collects several modules for working with the +:mod:`http` is a package that collects several modules for working with the HyperText Transfer Protocol: * :mod:`http.client` is a low-level HTTP protocol client; for high-level URL @@ -45,85 +45,79 @@ >>> list(HTTPStatus) [, , ...] - The supported HTTP status codes are: +.. _http-status-codes: - === ============================== - 100 Continue - 101 Switching Protocols - 102 Processing - 200 OK - 201 Created - 202 Accepted - 203 Non-Authoritative Information - 204 No Content - 205 Reset Content - 206 Partial Content - 207 Multi-Status - 208 Already Reported - 226 IM Used - 300 Multiple Choices - 301 Moved Permanently - 302 Found - 303 See Other - 304 Not Modified - 305 Use Proxy - 306 Switch Proxy - 307 Temporary Redirect - 308 Permanent Redirect - 400 Bad Request - 401 Unauthorized - 402 Payment Required - 403 Forbidden - 404 Not Found - 405 Method Not Allowed - 406 Not Acceptable - 407 Proxy Authentication Required - 408 Request Timeout - 409 Conflict - 410 Gone - 411 Length Required - 412 Precondition Failed - 413 Request Entity Too Large - 414 Request URI Too Long - 415 Unsupported Media Type - 416 Request Range Not Satisfiable - 417 Expectation Failed - 418 I'm a teapot - 419 Authentication Timeout - 420 Method Failure *(Spring framework)* - 422 Unprocessable Entity - 423 Locked - 424 Failed Dependency - 426 Upgrade Required - 428 Precondition Required - 429 Too Many Requests - 431 Request Header Field Too Large - 440 Login Timeout *(Microsoft)* - 444 No Response *(Nginx)* - 449 Retry With *(Microsoft)* - 450 Blocked By Windows Parental Controls *(Microsoft)* - 494 Request Header Too Large *(Nginx)* - 495 Cert Error *(Nginx)* - 496 No Cert *(Nginx)* - 497 HTTP To HTTPS *(Nginx)* - 499 Client Closed Request *(Nginx)* - 500 Internal Server Error - 501 Not Implemented - 502 Bad Gateway - 503 Service Unavailable - 504 Gateway Timeout - 505 HTTP Version Not Supported - 506 Variant Also Negotiates - 507 Insufficient Storage - 508 Loop Detected - 509 Bandwidth Limit Exceeded - 510 Not Extended - 511 Network Authentication Required - 520 Origin Error *(CloudFlare)* - 521 Web Server Is Down *(CloudFlare)* - 522 Connection Timed Out *(CloudFlare)* - 523 Proxy Declined Request *(CloudFlare)* - 524 A Timeout Occurred *(CloudFlare)* - 598 Network Read Timeout Error - 599 Network Connect Timeout Error - === ============================== +HTTP status codes +----------------- + +Supported, +`IANA-registered `_ +status codes available in :class:`http.HTTPStatus` are: + +======= =================================== ================================================================== +Code Enum Name Details +======= =================================== ================================================================== +``100`` ``CONTINUE`` HTTP/1.1 :rfc:`7231`, Section 6.2.1 +``101`` ``SWITCHING_PROTOCOLS`` HTTP/1.1 :rfc:`7231`, Section 6.2.2 +``102`` ``PROCESSING`` WebDAV :rfc:`2518`, Section 10.1 +``200`` ``OK`` HTTP/1.1 :rfc:`7231`, Section 6.3.1 +``201`` ``CREATED`` HTTP/1.1 :rfc:`7231`, Section 6.3.2 +``202`` ``ACCEPTED`` HTTP/1.1 :rfc:`7231`, Section 6.3.3 +``203`` ``NON_AUTHORITATIVE_INFORMATION`` HTTP/1.1 :rfc:`7231`, Section 6.3.4 +``204`` ``NO_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.5 +``205`` ``RESET_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.6 +``206`` ``PARTIAL_CONTENT`` HTTP/1.1 :rfc:`7233`, Section 4.1 +``207`` ``MULTI_STATUS`` WebDAV :rfc:`4918`, Section 11.1 +``208`` ``ALREADY_REPORTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.1 (Experimental) +``226`` ``IM_USED`` Delta Encoding in HTTP :rfc:`3229`, Section 10.4.1 +``300`` ``MULTIPLE_CHOICES`` HTTP/1.1 :rfc:`7231`, Section 6.4.1 +``301`` ``MOVED_PERMANENTLY`` HTTP/1.1 :rfc:`7231`, Section 6.4.2 +``302`` ``FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.4.3 +``303`` ``SEE_OTHER`` HTTP/1.1 :rfc:`7231`, Section 6.4.4 +``304`` ``NOT_MODIFIED`` HTTP/1.1 :rfc:`7232`, Section 4.1 +``305`` ``USE_PROXY`` HTTP/1.1 :rfc:`7231`, Section 6.4.5 +``307`` ``TEMPORARY_REDIRECT`` HTTP/1.1 :rfc:`7231`, Section 6.4.7 +``308`` ``PERMANENT_REDIRECT`` Permanent Redirect :rfc:`7238`, Section 3 (Experimental) +``400`` ``BAD_REQUEST`` HTTP/1.1 :rfc:`7231`, Section 6.5.1 +``401`` ``UNAUTHORIZED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.1 +``402`` ``PAYMENT_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.2 +``403`` ``FORBIDDEN`` HTTP/1.1 :rfc:`7231`, Section 6.5.3 +``404`` ``NOT_FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.5.4 +``405`` ``METHOD_NOT_ALLOWED`` HTTP/1.1 :rfc:`7231`, Section 6.5.5 +``406`` ``NOT_ACCEPTABLE`` HTTP/1.1 :rfc:`7231`, Section 6.5.6 +``407`` ``PROXY_AUTHENTICATION_REQUIRED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.2 +``408`` ``REQUEST_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.5.7 +``409`` ``CONFLICT`` HTTP/1.1 :rfc:`7231`, Section 6.5.8 +``410`` ``GONE`` HTTP/1.1 :rfc:`7231`, Section 6.5.9 +``411`` ``LENGTH_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.10 +``412`` ``PRECONDITION_FAILED`` HTTP/1.1 :rfc:`7232`, Section 4.2 +``413`` ``REQUEST_ENTITY_TOO_LARGE`` HTTP/1.1 :rfc:`7231`, Section 6.5.11 +``414`` ``REQUEST_URI_TOO_LONG`` HTTP/1.1 :rfc:`7231`, Section 6.5.12 +``415`` ``UNSUPPORTED_MEDIA_TYPE`` HTTP/1.1 :rfc:`7231`, Section 6.5.13 +``416`` ``REQUEST_RANGE_NOT_SATISFIABLE`` HTTP/1.1 Range Requests :rfc:`7233`, Section 4.4 +``417`` ``EXPECTATION_FAILED`` HTTP/1.1 :rfc:`7231`, Section 6.5.14 +``422`` ``UNPROCESSABLE_ENTITY`` WebDAV :rfc:`4918`, Section 11.2 +``423`` ``LOCKED`` WebDAV :rfc:`4918`, Section 11.3 +``424`` ``FAILED_DEPENDENCY`` WebDAV :rfc:`4918`, Section 11.4 +``426`` ``UPGRADE_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.15 +``428`` ``PRECONDITION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585` +``429`` ``TOO_MANY_REQUESTS`` Additional HTTP Status Codes :rfc:`6585` +``431`` ``REQUEST_HEADER_FIELDS_TOO_LARGE`` Additional HTTP Status Codes :rfc:`6585` +``500`` ``INTERNAL_SERVER_ERROR`` HTTP/1.1 :rfc:`7231`, Section 6.6.1 +``501`` ``NOT_IMPLEMENTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.2 +``502`` ``BAD_GATEWAY`` HTTP/1.1 :rfc:`7231`, Section 6.6.3 +``503`` ``SERVICE_UNAVAILABLE`` HTTP/1.1 :rfc:`7231`, Section 6.6.4 +``504`` ``GATEWAY_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.6.5 +``505`` ``HTTP_VERSION_NOT_SUPPORTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.6 +``506`` ``VARIANT_ALSO_NEGOTIATES`` Transparent Content Negotiation in HTTP :rfc:`2295`, Section 8.1 (Experimental) +``507`` ``INSUFFICIENT_STORAGE`` WebDAV :rfc:`4918`, Section 11.5 +``508`` ``LOOP_DETECTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.2 (Experimental) +``510`` ``NOT_EXTENDED`` An HTTP Extension Framework :rfc:`2774`, Section 7 (Experimental) +``511`` ``NETWORK_AUTHENTICATION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585`, Section 6 +======= =================================== ================================================================== + +In order to preserve backwards compatibility, enum values are also present +in the :mod:`http.client` and :mod:`http.server` modules in the form of +constants. The enum name is equal to the constant name (i.e. +``http.HTTPStatus.OK`` is also available as ``http.client.OK`` and +``http.server.OK``). diff --git a/Lib/http/__init__.py b/Lib/http/__init__.py --- a/Lib/http/__init__.py +++ b/Lib/http/__init__.py @@ -13,16 +13,8 @@ * RFC 4918: HTTP Extensions for WebDAV, obsoletes 2518 * RFC 5842: Binding Extensions to WebDAV * RFC 7238: Permanent Redirect - * RFC 2324: Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0) * RFC 2295: Transparent Content Negotiation in HTTP * RFC 2774: An HTTP Extension Framework - - Non-standard vendor codes include: - - * Spring framework: 420 - * Nginx: 444, 494, 495, 496, 497, 499 - * Microsoft: 440, 449, 450 - * Cloudflare: 520, 521, 522, 523, 524, 598, 599 """ def __new__(cls, value, phrase, description=''): obj = int.__new__(cls, value) @@ -63,7 +55,6 @@ 'Document has not changed since given time') USE_PROXY = (305, 'Use Proxy', 'You must use proxy specified in Location to access this resource') - SWITCH_PROXY = 306, 'Switch Proxy' TEMPORARY_REDIRECT = (307, 'Temporary Redirect', 'Object moved temporarily -- see URI list') PERMANENT_REDIRECT = (308, 'Permanent Redirect', @@ -107,9 +98,6 @@ 'Cannot satisfy request range') EXPECTATION_FAILED = (417, 'Expectation Failed', 'Expect condition could not be satisfied') - IM_A_TEAPOT = 418, 'I\'m a teapot' - AUTHENTICATION_TIMEOUT = 419, 'Authentication Timeout' - METHOD_FAILURE = 420, 'Method Failure' # Spring framework UNPROCESSABLE_ENTITY = 422, 'Unprocessable Entity' LOCKED = 423, 'Locked' FAILED_DEPENDENCY = 424, 'Failed Dependency' @@ -123,16 +111,6 @@ 'Request Header Field Too Large', 'The server is unwilling to process the request because its header ' 'fields are too large') - LOGIN_TIMEOUT = 440, 'Login Timeout' # microsoft - NO_RESPONSE = 444, 'No Response' # nginx - RETRY_WITH = 449, 'Retry With' # microsoft - BLOCKED_BY_WINDOWS_PARENTAL_CONTROLS = (450, - 'Blocked By Windows Parental Controls') # microsoft - REQUEST_HEADER_TOO_LARGE = 494, 'Request Header Too Large' # nginx - CERT_ERROR = 495, 'Cert Error' # nginx - NO_CERT = 496, 'No Cert' # nginx - HTTP_TO_HTTPS = 497, 'HTTP To HTTPS' # nginx - CLIENT_CLOSED_REQUEST = 499, 'Client Closed Request' # nginx # server errors INTERNAL_SERVER_ERROR = (500, 'Internal Server Error', @@ -150,15 +128,7 @@ VARIANT_ALSO_NEGOTIATES = 506, 'Variant Also Negotiates' INSUFFICIENT_STORAGE = 507, 'Insufficient Storage' LOOP_DETECTED = 508, 'Loop Detected' - BANDWIDTH_LIMIT_EXCEEDED = 509, 'Bandwidth Limit Exceeded' NOT_EXTENDED = 510, 'Not Extended' NETWORK_AUTHENTICATION_REQUIRED = (511, 'Network Authentication Required', 'The client needs to authenticate to gain network access') - ORIGIN_ERROR = 520, 'Origin Error' # cloudflare - WEB_SERVER_IS_DOWN = 521, 'Web Server Is Down' # cloudflare - CONNECTON_TIMED_OUT = 522, 'Connection Timed Out' # cloudflare - PROXY_DECLINED_REQUEST = 523, 'Proxy Declined Request' # cloudflare - A_TIMEOUT_OCCURRED = 524, 'A Timeout Occurred', '' # cloudflare - NETWORK_READ_TIMEOUT_ERROR = 598, 'Network Read Timeout Error' - NETWORK_CONNECT_TIMEOUT_ERROR = 599, 'Network Connect Timeout Error' -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 20 05:46:45 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 20 Jan 2015 04:46:45 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyMzE3?= =?utf-8?q?=3A_Document_the_action_parameter_in_ArgumentParser=2Eadd=5Fsub?= =?utf-8?b?cGFyc2Vycygp?= Message-ID: <20150120044642.103287.56823@psf.io> https://hg.python.org/cpython/rev/350b8e109c42 changeset: 94220:350b8e109c42 branch: 3.4 parent: 94209:2eea364c2863 user: Berker Peksag date: Tue Jan 20 06:45:53 2015 +0200 summary: Issue #22317: Document the action parameter in ArgumentParser.add_subparsers() docs. Patch by Mike Short. files: Doc/library/argparse.rst | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1521,12 +1521,15 @@ * parser_class - class which will be used to create sub-parser instances, by default the class of the current parser (e.g. ArgumentParser) - * dest - name of the attribute under which sub-command name will be + * action_ - the basic type of action to be taken when this argument is + encountered at the command line + + * dest_ - name of the attribute under which sub-command name will be stored; by default None and no value is stored - * help - help for sub-parser group in help output, by default None + * help_ - help for sub-parser group in help output, by default None - * metavar - string presenting available sub-commands in help; by default it + * metavar_ - string presenting available sub-commands in help; by default it is None and presents sub-commands in form {cmd1, cmd2, ..} Some example usage:: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 20 05:46:45 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 20 Jan 2015 04:46:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322317=3A_Document_the_action_parameter_in_Argum?= =?utf-8?q?entParser=2Eadd=5Fsubparsers=28=29?= Message-ID: <20150120044643.69918.23713@psf.io> https://hg.python.org/cpython/rev/4709290253e3 changeset: 94221:4709290253e3 parent: 94219:c8647dab4780 parent: 94220:350b8e109c42 user: Berker Peksag date: Tue Jan 20 06:46:49 2015 +0200 summary: Issue #22317: Document the action parameter in ArgumentParser.add_subparsers() docs. Patch by Mike Short. files: Doc/library/argparse.rst | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1521,12 +1521,15 @@ * parser_class - class which will be used to create sub-parser instances, by default the class of the current parser (e.g. ArgumentParser) - * dest - name of the attribute under which sub-command name will be + * action_ - the basic type of action to be taken when this argument is + encountered at the command line + + * dest_ - name of the attribute under which sub-command name will be stored; by default None and no value is stored - * help - help for sub-parser group in help output, by default None + * help_ - help for sub-parser group in help output, by default None - * metavar - string presenting available sub-commands in help; by default it + * metavar_ - string presenting available sub-commands in help; by default it is None and presents sub-commands in form {cmd1, cmd2, ..} Some example usage:: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 20 05:55:46 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 20 Jan 2015 04:55:46 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIyMzE3?= =?utf-8?q?=3A_Document_the_action_parameter_in_ArgumentParser=2Eadd=5Fsub?= =?utf-8?b?cGFyc2Vycygp?= Message-ID: <20150120045542.103289.33483@psf.io> https://hg.python.org/cpython/rev/430236ef507b changeset: 94222:430236ef507b branch: 2.7 parent: 94211:6e03c18f5447 user: Berker Peksag date: Tue Jan 20 06:55:51 2015 +0200 summary: Issue #22317: Document the action parameter in ArgumentParser.add_subparsers() docs. Patch by Mike Short. files: Doc/library/argparse.rst | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1499,12 +1499,15 @@ * parser_class - class which will be used to create sub-parser instances, by default the class of the current parser (e.g. ArgumentParser) - * dest - name of the attribute under which sub-command name will be + * action_ - the basic type of action to be taken when this argument is + encountered at the command line + + * dest_ - name of the attribute under which sub-command name will be stored; by default None and no value is stored - * help - help for sub-parser group in help output, by default None + * help_ - help for sub-parser group in help output, by default None - * metavar - string presenting available sub-commands in help; by default it + * metavar_ - string presenting available sub-commands in help; by default it is None and presents sub-commands in form {cmd1, cmd2, ..} Some example usage:: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 20 07:02:29 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 20 Jan 2015 06:02:29 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320898=3A_Enum_nam?= =?utf-8?q?es_are_only_available_in_the_http=2Eclient_module_as?= Message-ID: <20150120060220.93183.45421@psf.io> https://hg.python.org/cpython/rev/3a95a74aca4e changeset: 94223:3a95a74aca4e parent: 94221:4709290253e3 user: Berker Peksag date: Tue Jan 20 08:02:28 2015 +0200 summary: Issue #20898: Enum names are only available in the http.client module as constants. Noticed by Martin Panter. files: Doc/library/http.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/http.rst b/Doc/library/http.rst --- a/Doc/library/http.rst +++ b/Doc/library/http.rst @@ -117,7 +117,6 @@ ======= =================================== ================================================================== In order to preserve backwards compatibility, enum values are also present -in the :mod:`http.client` and :mod:`http.server` modules in the form of -constants. The enum name is equal to the constant name (i.e. -``http.HTTPStatus.OK`` is also available as ``http.client.OK`` and -``http.server.OK``). +in the :mod:`http.client` module in the form of constants. The enum name is +equal to the constant name (i.e. ``http.HTTPStatus.OK`` is also available as +``http.client.OK``). -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Tue Jan 20 09:34:29 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 20 Jan 2015 09:34:29 +0100 Subject: [Python-checkins] Daily reference leaks (ac0d6c09457e): sum=3 Message-ID: results for ac0d6c09457e on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloglgm2QK', '-x'] From python-checkins at python.org Tue Jan 20 21:12:41 2015 From: python-checkins at python.org (zach.ware) Date: Tue, 20 Jan 2015 20:12:41 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2Ugd2l0aCAzLjQgKGNsb3NlcyAjMjMyODAp?= Message-ID: <20150120201215.103295.50238@psf.io> https://hg.python.org/cpython/rev/754c630c98a3 changeset: 94225:754c630c98a3 parent: 94223:3a95a74aca4e parent: 94224:1cb2b46c5109 user: Zachary Ware date: Tue Jan 20 14:11:38 2015 -0600 summary: Merge with 3.4 (closes #23280) files: Lib/test/test_binascii.py | 4 +- Modules/binascii.c | 35 +++++++++++- Modules/clinic/binascii.c.h | 70 ++++++++++++++++++++++++- 3 files changed, 103 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py --- a/Lib/test/test_binascii.py +++ b/Lib/test/test_binascii.py @@ -162,7 +162,9 @@ self.assertRaises(binascii.Error, binascii.a2b_hex, t[:-1]) self.assertRaises(binascii.Error, binascii.a2b_hex, t[:-1] + b'q') - self.assertEqual(binascii.hexlify(b'a'), b'61') + # Confirm that b2a_hex == hexlify and a2b_hex == unhexlify + self.assertEqual(binascii.hexlify(self.type2test(s)), t) + self.assertEqual(binascii.unhexlify(self.type2test(t)), u) def test_qp(self): # A test for SF bug 534347 (segfaults without the proper fix) diff --git a/Modules/binascii.c b/Modules/binascii.c --- a/Modules/binascii.c +++ b/Modules/binascii.c @@ -1147,6 +1147,20 @@ return retval; } +/*[clinic input] +binascii.hexlify = binascii.b2a_hex + +Hexadecimal representation of binary data. + +The return value is a bytes object. +[clinic start generated code]*/ + +static PyObject * +binascii_hexlify_impl(PyModuleDef *module, Py_buffer *data) +/*[clinic end generated code: output=6098440091fb61dc input=2e3afae7f083f061]*/ +{ + return binascii_b2a_hex_impl(module, data); +} static int to_int(int c) @@ -1221,6 +1235,21 @@ return NULL; } +/*[clinic input] +binascii.unhexlify = binascii.a2b_hex + +Binary data of hexadecimal representation. + +hexstr must contain an even number of hex digits (upper or lower case). +[clinic start generated code]*/ + +static PyObject * +binascii_unhexlify_impl(PyModuleDef *module, Py_buffer *hexstr) +/*[clinic end generated code: output=17cec7544499803e input=dd8c012725f462da]*/ +{ + return binascii_a2b_hex_impl(module, hexstr); +} + static int table_hex[128] = { -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, @@ -1535,10 +1564,8 @@ BINASCII_B2A_HQX_METHODDEF BINASCII_A2B_HEX_METHODDEF BINASCII_B2A_HEX_METHODDEF - {"unhexlify", (PyCFunction)binascii_a2b_hex, METH_VARARGS, - binascii_a2b_hex__doc__}, - {"hexlify", (PyCFunction)binascii_b2a_hex, METH_VARARGS, - binascii_b2a_hex__doc__}, + BINASCII_HEXLIFY_METHODDEF + BINASCII_UNHEXLIFY_METHODDEF BINASCII_RLECODE_HQX_METHODDEF BINASCII_RLEDECODE_HQX_METHODDEF BINASCII_CRC_HQX_METHODDEF diff --git a/Modules/clinic/binascii.c.h b/Modules/clinic/binascii.c.h --- a/Modules/clinic/binascii.c.h +++ b/Modules/clinic/binascii.c.h @@ -367,6 +367,40 @@ return return_value; } +PyDoc_STRVAR(binascii_hexlify__doc__, +"hexlify($module, data, /)\n" +"--\n" +"\n" +"Hexadecimal representation of binary data.\n" +"\n" +"The return value is a bytes object."); + +#define BINASCII_HEXLIFY_METHODDEF \ + {"hexlify", (PyCFunction)binascii_hexlify, METH_VARARGS, binascii_hexlify__doc__}, + +static PyObject * +binascii_hexlify_impl(PyModuleDef *module, Py_buffer *data); + +static PyObject * +binascii_hexlify(PyModuleDef *module, PyObject *args) +{ + PyObject *return_value = NULL; + Py_buffer data = {NULL, NULL}; + + if (!PyArg_ParseTuple(args, + "y*:hexlify", + &data)) + goto exit; + return_value = binascii_hexlify_impl(module, &data); + +exit: + /* Cleanup for data */ + if (data.obj) + PyBuffer_Release(&data); + + return return_value; +} + PyDoc_STRVAR(binascii_a2b_hex__doc__, "a2b_hex($module, hexstr, /)\n" "--\n" @@ -402,6 +436,40 @@ return return_value; } +PyDoc_STRVAR(binascii_unhexlify__doc__, +"unhexlify($module, hexstr, /)\n" +"--\n" +"\n" +"Binary data of hexadecimal representation.\n" +"\n" +"hexstr must contain an even number of hex digits (upper or lower case)."); + +#define BINASCII_UNHEXLIFY_METHODDEF \ + {"unhexlify", (PyCFunction)binascii_unhexlify, METH_VARARGS, binascii_unhexlify__doc__}, + +static PyObject * +binascii_unhexlify_impl(PyModuleDef *module, Py_buffer *hexstr); + +static PyObject * +binascii_unhexlify(PyModuleDef *module, PyObject *args) +{ + PyObject *return_value = NULL; + Py_buffer hexstr = {NULL, NULL}; + + if (!PyArg_ParseTuple(args, + "O&:unhexlify", + ascii_buffer_converter, &hexstr)) + goto exit; + return_value = binascii_unhexlify_impl(module, &hexstr); + +exit: + /* Cleanup for hexstr */ + if (hexstr.obj) + PyBuffer_Release(&hexstr); + + return return_value; +} + PyDoc_STRVAR(binascii_a2b_qp__doc__, "a2b_qp($module, /, data, header=False)\n" "--\n" @@ -475,4 +543,4 @@ return return_value; } -/*[clinic end generated code: output=53cd6b379c745220 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=771126f8f53e84e7 input=a9049054013a1b77]*/ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 20 21:12:41 2015 From: python-checkins at python.org (zach.ware) Date: Tue, 20 Jan 2015 20:12:41 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjgw?= =?utf-8?q?=3A_Fix_docstrings_for_binascii=2E=28un=29hexlify?= Message-ID: <20150120201215.118074.20309@psf.io> https://hg.python.org/cpython/rev/1cb2b46c5109 changeset: 94224:1cb2b46c5109 branch: 3.4 parent: 94220:350b8e109c42 user: Zachary Ware date: Tue Jan 20 13:59:46 2015 -0600 summary: Issue #23280: Fix docstrings for binascii.(un)hexlify files: Lib/test/test_binascii.py | 4 +- Modules/binascii.c | 35 +++++++++++- Modules/clinic/binascii.c.h | 70 ++++++++++++++++++++++++- 3 files changed, 103 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py --- a/Lib/test/test_binascii.py +++ b/Lib/test/test_binascii.py @@ -162,7 +162,9 @@ self.assertRaises(binascii.Error, binascii.a2b_hex, t[:-1]) self.assertRaises(binascii.Error, binascii.a2b_hex, t[:-1] + b'q') - self.assertEqual(binascii.hexlify(b'a'), b'61') + # Confirm that b2a_hex == hexlify and a2b_hex == unhexlify + self.assertEqual(binascii.hexlify(self.type2test(s)), t) + self.assertEqual(binascii.unhexlify(self.type2test(t)), u) def test_qp(self): # A test for SF bug 534347 (segfaults without the proper fix) diff --git a/Modules/binascii.c b/Modules/binascii.c --- a/Modules/binascii.c +++ b/Modules/binascii.c @@ -1147,6 +1147,20 @@ return retval; } +/*[clinic input] +binascii.hexlify = binascii.b2a_hex + +Hexadecimal representation of binary data. + +The return value is a bytes object. +[clinic start generated code]*/ + +static PyObject * +binascii_hexlify_impl(PyModuleDef *module, Py_buffer *data) +/*[clinic end generated code: output=6098440091fb61dc input=2e3afae7f083f061]*/ +{ + return binascii_b2a_hex_impl(module, data); +} static int to_int(int c) @@ -1221,6 +1235,21 @@ return NULL; } +/*[clinic input] +binascii.unhexlify = binascii.a2b_hex + +Binary data of hexadecimal representation. + +hexstr must contain an even number of hex digits (upper or lower case). +[clinic start generated code]*/ + +static PyObject * +binascii_unhexlify_impl(PyModuleDef *module, Py_buffer *hexstr) +/*[clinic end generated code: output=17cec7544499803e input=dd8c012725f462da]*/ +{ + return binascii_a2b_hex_impl(module, hexstr); +} + static int table_hex[128] = { -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, @@ -1535,10 +1564,8 @@ BINASCII_B2A_HQX_METHODDEF BINASCII_A2B_HEX_METHODDEF BINASCII_B2A_HEX_METHODDEF - {"unhexlify", (PyCFunction)binascii_a2b_hex, METH_VARARGS, - binascii_a2b_hex__doc__}, - {"hexlify", (PyCFunction)binascii_b2a_hex, METH_VARARGS, - binascii_b2a_hex__doc__}, + BINASCII_HEXLIFY_METHODDEF + BINASCII_UNHEXLIFY_METHODDEF BINASCII_RLECODE_HQX_METHODDEF BINASCII_RLEDECODE_HQX_METHODDEF BINASCII_CRC_HQX_METHODDEF diff --git a/Modules/clinic/binascii.c.h b/Modules/clinic/binascii.c.h --- a/Modules/clinic/binascii.c.h +++ b/Modules/clinic/binascii.c.h @@ -367,6 +367,40 @@ return return_value; } +PyDoc_STRVAR(binascii_hexlify__doc__, +"hexlify($module, data, /)\n" +"--\n" +"\n" +"Hexadecimal representation of binary data.\n" +"\n" +"The return value is a bytes object."); + +#define BINASCII_HEXLIFY_METHODDEF \ + {"hexlify", (PyCFunction)binascii_hexlify, METH_VARARGS, binascii_hexlify__doc__}, + +static PyObject * +binascii_hexlify_impl(PyModuleDef *module, Py_buffer *data); + +static PyObject * +binascii_hexlify(PyModuleDef *module, PyObject *args) +{ + PyObject *return_value = NULL; + Py_buffer data = {NULL, NULL}; + + if (!PyArg_ParseTuple(args, + "y*:hexlify", + &data)) + goto exit; + return_value = binascii_hexlify_impl(module, &data); + +exit: + /* Cleanup for data */ + if (data.obj) + PyBuffer_Release(&data); + + return return_value; +} + PyDoc_STRVAR(binascii_a2b_hex__doc__, "a2b_hex($module, hexstr, /)\n" "--\n" @@ -402,6 +436,40 @@ return return_value; } +PyDoc_STRVAR(binascii_unhexlify__doc__, +"unhexlify($module, hexstr, /)\n" +"--\n" +"\n" +"Binary data of hexadecimal representation.\n" +"\n" +"hexstr must contain an even number of hex digits (upper or lower case)."); + +#define BINASCII_UNHEXLIFY_METHODDEF \ + {"unhexlify", (PyCFunction)binascii_unhexlify, METH_VARARGS, binascii_unhexlify__doc__}, + +static PyObject * +binascii_unhexlify_impl(PyModuleDef *module, Py_buffer *hexstr); + +static PyObject * +binascii_unhexlify(PyModuleDef *module, PyObject *args) +{ + PyObject *return_value = NULL; + Py_buffer hexstr = {NULL, NULL}; + + if (!PyArg_ParseTuple(args, + "O&:unhexlify", + ascii_buffer_converter, &hexstr)) + goto exit; + return_value = binascii_unhexlify_impl(module, &hexstr); + +exit: + /* Cleanup for hexstr */ + if (hexstr.obj) + PyBuffer_Release(&hexstr); + + return return_value; +} + PyDoc_STRVAR(binascii_a2b_qp__doc__, "a2b_qp($module, /, data, header=False)\n" "--\n" @@ -475,4 +543,4 @@ return return_value; } -/*[clinic end generated code: output=68e2bcc6956b6213 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=e46d29f8c9adae7e input=a9049054013a1b77]*/ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 21 02:22:22 2015 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 21 Jan 2015 01:22:22 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Add_the_comman?= =?utf-8?q?d_line_to_the_AssertionError_raised_by_test=2Escript=5Fhelper?= =?utf-8?q?=27s?= Message-ID: <20150121012206.103289.32872@psf.io> https://hg.python.org/cpython/rev/d1ed573a2fa3 changeset: 94226:d1ed573a2fa3 branch: 3.4 parent: 94224:1cb2b46c5109 user: Gregory P. Smith date: Tue Jan 20 17:19:47 2015 -0800 summary: Add the command line to the AssertionError raised by test.script_helper's Python subprocess failure assertion error messages for easier debugging. Adds a unittest for test.script_helper to confirm that this code works as it is otherwise uncovered by an already passing test suite that uses it. :) files: Lib/test/script_helper.py | 5 +- Lib/test/test_script_helper.py | 35 ++++++++++++++++++++++ 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py --- a/Lib/test/script_helper.py +++ b/Lib/test/script_helper.py @@ -51,8 +51,9 @@ err = strip_python_stderr(err) if (rc and expected_success) or (not rc and not expected_success): raise AssertionError( - "Process return code is %d, " - "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore'))) + "Process return code is %d, command line was: %r, " + "stderr follows:\n%s" % (rc, cmd_line, + err.decode('ascii', 'ignore'))) return rc, out, err def assert_python_ok(*args, **env_vars): diff --git a/Lib/test/test_script_helper.py b/Lib/test/test_script_helper.py new file mode 100755 --- /dev/null +++ b/Lib/test/test_script_helper.py @@ -0,0 +1,35 @@ +"""Unittests for test.script_helper. Who tests the test helper?""" + +from test import script_helper +import unittest + + +class TestScriptHelper(unittest.TestCase): + def test_assert_python_expect_success(self): + t = script_helper._assert_python(True, '-c', 'import sys; sys.exit(0)') + self.assertEqual(0, t[0], 'return code was not 0') + + def test_assert_python_expect_failure(self): + # I didn't import the sys module so this child will fail. + rc, out, err = script_helper._assert_python(False, '-c', 'sys.exit(0)') + self.assertNotEqual(0, rc, 'return code should not be 0') + + def test_assert_python_raises_expect_success(self): + # I didn't import the sys module so this child will fail. + with self.assertRaises(AssertionError) as error_context: + script_helper._assert_python(True, '-c', 'sys.exit(0)') + error_msg = str(error_context.exception) + self.assertIn('command line was:', error_msg) + self.assertIn('sys.exit(0)', error_msg, msg='unexpected command line') + + def test_assert_python_raises_expect_failure(self): + with self.assertRaises(AssertionError) as error_context: + script_helper._assert_python(False, '-c', 'import sys; sys.exit(0)') + error_msg = str(error_context.exception) + self.assertIn('Process return code is 0,', error_msg) + self.assertIn('import sys; sys.exit(0)', error_msg, + msg='unexpected command line.') + + +if __name__ == '__main__': + unittest.main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 21 02:22:22 2015 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 21 Jan 2015 01:22:22 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Add_the_command_line_to_the_AssertionError_raised_by_tes?= =?utf-8?q?t=2Escript=5Fhelper=27s?= Message-ID: <20150121012206.69920.75845@psf.io> https://hg.python.org/cpython/rev/0893b9ee44ea changeset: 94227:0893b9ee44ea parent: 94225:754c630c98a3 parent: 94226:d1ed573a2fa3 user: Gregory P. Smith date: Tue Jan 20 17:21:41 2015 -0800 summary: Add the command line to the AssertionError raised by test.script_helper's Python subprocess failure assertion error messages for easier debugging. Adds a unittest for test.script_helper to confirm that this code works as it is otherwise uncovered by an already passing test suite that uses it. :) files: Lib/test/script_helper.py | 5 +- Lib/test/test_script_helper.py | 35 ++++++++++++++++++++++ 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py --- a/Lib/test/script_helper.py +++ b/Lib/test/script_helper.py @@ -51,8 +51,9 @@ err = strip_python_stderr(err) if (rc and expected_success) or (not rc and not expected_success): raise AssertionError( - "Process return code is %d, " - "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore'))) + "Process return code is %d, command line was: %r, " + "stderr follows:\n%s" % (rc, cmd_line, + err.decode('ascii', 'ignore'))) return rc, out, err def assert_python_ok(*args, **env_vars): diff --git a/Lib/test/test_script_helper.py b/Lib/test/test_script_helper.py new file mode 100755 --- /dev/null +++ b/Lib/test/test_script_helper.py @@ -0,0 +1,35 @@ +"""Unittests for test.script_helper. Who tests the test helper?""" + +from test import script_helper +import unittest + + +class TestScriptHelper(unittest.TestCase): + def test_assert_python_expect_success(self): + t = script_helper._assert_python(True, '-c', 'import sys; sys.exit(0)') + self.assertEqual(0, t[0], 'return code was not 0') + + def test_assert_python_expect_failure(self): + # I didn't import the sys module so this child will fail. + rc, out, err = script_helper._assert_python(False, '-c', 'sys.exit(0)') + self.assertNotEqual(0, rc, 'return code should not be 0') + + def test_assert_python_raises_expect_success(self): + # I didn't import the sys module so this child will fail. + with self.assertRaises(AssertionError) as error_context: + script_helper._assert_python(True, '-c', 'sys.exit(0)') + error_msg = str(error_context.exception) + self.assertIn('command line was:', error_msg) + self.assertIn('sys.exit(0)', error_msg, msg='unexpected command line') + + def test_assert_python_raises_expect_failure(self): + with self.assertRaises(AssertionError) as error_context: + script_helper._assert_python(False, '-c', 'import sys; sys.exit(0)') + error_msg = str(error_context.exception) + self.assertIn('Process return code is 0,', error_msg) + self.assertIn('import sys; sys.exit(0)', error_msg, + msg='unexpected command line.') + + +if __name__ == '__main__': + unittest.main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 21 06:48:01 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 21 Jan 2015 05:48:01 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_detect_64-bit_?= =?utf-8?q?systems_using_maxsize_not_maxint?= Message-ID: <20150121054758.118088.13658@psf.io> https://hg.python.org/cpython/rev/2cd0aa44d53c changeset: 94228:2cd0aa44d53c branch: 2.7 parent: 94222:430236ef507b user: Benjamin Peterson date: Wed Jan 21 00:47:54 2015 -0500 summary: detect 64-bit systems using maxsize not maxint files: setup.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -696,7 +696,7 @@ exts.append( Extension('audioop', ['audioop.c']) ) # Disabled on 64-bit platforms - if sys.maxint != 9223372036854775807L: + if sys.maxsize != 9223372036854775807L: # Operations on images exts.append( Extension('imageop', ['imageop.c']) ) else: -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Wed Jan 21 09:47:09 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 21 Jan 2015 09:47:09 +0100 Subject: [Python-checkins] Daily reference leaks (0893b9ee44ea): sum=3 Message-ID: results for 0893b9ee44ea on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloghEYoVG', '-x'] From python-checkins at python.org Wed Jan 21 23:41:53 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 21 Jan 2015 22:41:53 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRW5o?= =?utf-8?q?ance_BaseProactorEventLoop=2E=5Floop=5Fself=5Freading=28=29?= Message-ID: <20150121224133.104120.36475@psf.io> https://hg.python.org/cpython/rev/d36d60d19095 changeset: 94229:d36d60d19095 branch: 3.4 parent: 94226:d1ed573a2fa3 user: Victor Stinner date: Wed Jan 21 23:38:37 2015 +0100 summary: asyncio: Enhance BaseProactorEventLoop._loop_self_reading() * Handle correctly CancelledError: just exit * On error, log the exception and exit Don't try to close the event loop, it is probably running and so it cannot be closed. files: Lib/asyncio/proactor_events.py | 12 +++++++-- Lib/test/test_asyncio/test_proactor_events.py | 5 ++- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -463,9 +463,15 @@ if f is not None: f.result() # may raise f = self._proactor.recv(self._ssock, 4096) - except: - self.close() - raise + except futures.CancelledError: + # _close_self_pipe() has been called, stop waiting for data + return + except Exception as exc: + self.call_exception_handler({ + 'message': 'Error on reading from the event loop self pipe', + 'exception': exc, + 'loop': self, + }) else: self._self_reading_future = f f.add_done_callback(self._loop_self_reading) diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -523,9 +523,10 @@ def test_loop_self_reading_exception(self): self.loop.close = mock.Mock() + self.loop.call_exception_handler = mock.Mock() self.proactor.recv.side_effect = OSError() - self.assertRaises(OSError, self.loop._loop_self_reading) - self.assertTrue(self.loop.close.called) + self.loop._loop_self_reading() + self.assertTrue(self.loop.call_exception_handler.called) def test_write_to_self(self): self.loop._write_to_self() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 21 23:41:53 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 21 Jan 2015 22:41:53 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogcHlm?= =?utf-8?q?lakes=2C_remove_unused_import?= Message-ID: <20150121224133.93177.96796@psf.io> https://hg.python.org/cpython/rev/cf91f281fb9f changeset: 94230:cf91f281fb9f branch: 3.4 user: Victor Stinner date: Wed Jan 21 23:39:16 2015 +0100 summary: asyncio: pyflakes, remove unused import tests: Remove unused function; inline another function files: Lib/asyncio/base_events.py | 2 +- Lib/asyncio/selector_events.py | 1 - Lib/test/test_asyncio/test_streams.py | 10 +--------- 3 files changed, 2 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -636,7 +636,7 @@ try: yield from waiter - except Exception as exc: + except Exception: transport.close() raise diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -10,7 +10,6 @@ import errno import functools import socket -import sys try: import ssl except ImportError: # pragma: no cover diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -415,10 +415,6 @@ def set_err(): stream.set_exception(ValueError()) - @asyncio.coroutine - def readline(): - yield from stream.readline() - t1 = asyncio.Task(stream.readline(), loop=self.loop) t2 = asyncio.Task(set_err(), loop=self.loop) @@ -429,11 +425,7 @@ def test_exception_cancel(self): stream = asyncio.StreamReader(loop=self.loop) - @asyncio.coroutine - def read_a_line(): - yield from stream.readline() - - t = asyncio.Task(read_a_line(), loop=self.loop) + t = asyncio.Task(stream.readline(), loop=self.loop) test_utils.run_briefly(self.loop) t.cancel() test_utils.run_briefly(self.loop) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 21 23:41:53 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 21 Jan 2015 22:41:53 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDk1?= =?utf-8?q?=2C_asyncio=3A_Rewrite_=5FWaitHandleFuture=2Ecancel=28=29?= Message-ID: <20150121224133.103309.44497@psf.io> https://hg.python.org/cpython/rev/fb8a093db8b1 changeset: 94231:fb8a093db8b1 branch: 3.4 user: Victor Stinner date: Wed Jan 21 23:39:51 2015 +0100 summary: Issue #23095, asyncio: Rewrite _WaitHandleFuture.cancel() This change fixes a race conditon related to _WaitHandleFuture.cancel() leading to Python crash or "GetQueuedCompletionStatus() returned an unexpected event" logs. Before, the overlapped object was destroyed too early, it was possible that the wait completed whereas the overlapped object was already destroyed. Sometimes, a different overlapped was allocated at the same address, leading to unexpected completition. _WaitHandleFuture.cancel() now waits until the wait is cancelled to clear its reference to the overlapped object. To wait until the cancellation is done, UnregisterWaitEx() is used with an event instead of UnregisterWait(). To wait for this event, a new _WaitCancelFuture class was added. It's a simplified version of _WaitCancelFuture. For example, its cancel() method calls UnregisterWait(), not UnregisterWaitEx(). _WaitCancelFuture should not be cancelled. The overlapped object is kept alive in _WaitHandleFuture until the wait is unregistered. Other changes: * Add _overlapped.UnregisterWaitEx() * Remove fast-path in IocpProactor.wait_for_handle() to immediatly set the result if the wait already completed. I'm not sure that it's safe to call immediatly UnregisterWaitEx() before the completion was signaled. * Add IocpProactor._unregistered() to forget an overlapped which may never be signaled, but may be signaled for the next loop iteration. It avoids to block forever IocpProactor.close() if a wait was cancelled, and it may also avoid some "... unexpected event ..." warnings. files: Lib/asyncio/windows_events.py | 168 +++++++++++++++++---- Modules/overlapped.c | 25 +++ 2 files changed, 159 insertions(+), 34 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -78,20 +78,23 @@ self._ov = None -class _WaitHandleFuture(futures.Future): +class _BaseWaitHandleFuture(futures.Future): """Subclass of Future which represents a wait handle.""" - def __init__(self, iocp, ov, handle, wait_handle, *, loop=None): + def __init__(self, ov, handle, wait_handle, *, loop=None): super().__init__(loop=loop) if self._source_traceback: del self._source_traceback[-1] - # iocp and ov are only used by cancel() to notify IocpProactor - # that the wait was cancelled - self._iocp = iocp + # Keep a reference to the Overlapped object to keep it alive until the + # wait is unregistered self._ov = ov self._handle = handle self._wait_handle = wait_handle + # Should we call UnregisterWaitEx() if the wait completes + # or is cancelled? + self._registered = True + def _poll(self): # non-blocking wait: use a timeout of 0 millisecond return (_winapi.WaitForSingleObject(self._handle, 0) == @@ -99,21 +102,32 @@ def _repr_info(self): info = super()._repr_info() - info.insert(1, 'handle=%#x' % self._handle) - if self._wait_handle: + info.append('handle=%#x' % self._handle) + if self._handle is not None: state = 'signaled' if self._poll() else 'waiting' - info.insert(1, 'wait_handle=<%s, %#x>' - % (state, self._wait_handle)) + info.append(state) + if self._wait_handle is not None: + info.append('wait_handle=%#x' % self._wait_handle) return info + def _unregister_wait_cb(self, fut): + # The wait was unregistered: it's not safe to destroy the Overlapped + # object + self._ov = None + def _unregister_wait(self): - if self._wait_handle is None: + if not self._registered: return + self._registered = False + try: _overlapped.UnregisterWait(self._wait_handle) except OSError as exc: - # ERROR_IO_PENDING is not an error, the wait was unregistered - if exc.winerror != _overlapped.ERROR_IO_PENDING: + self._wait_handle = None + if exc.winerror == _overlapped.ERROR_IO_PENDING: + # ERROR_IO_PENDING is not an error, the wait was unregistered + self._unregister_wait_cb(None) + elif exc.winerror != _overlapped.ERROR_IO_PENDING: context = { 'message': 'Failed to unregister the wait handle', 'exception': exc, @@ -122,26 +136,91 @@ if self._source_traceback: context['source_traceback'] = self._source_traceback self._loop.call_exception_handler(context) - self._wait_handle = None - self._iocp = None - self._ov = None + else: + self._wait_handle = None + self._unregister_wait_cb(None) def cancel(self): - result = super().cancel() - if self._ov is not None: - # signal the cancellation to the overlapped object - _overlapped.PostQueuedCompletionStatus(self._iocp, True, - 0, self._ov.address) self._unregister_wait() - return result + return super().cancel() def set_exception(self, exception): + self._unregister_wait() super().set_exception(exception) - self._unregister_wait() def set_result(self, result): + self._unregister_wait() super().set_result(result) - self._unregister_wait() + + +class _WaitCancelFuture(_BaseWaitHandleFuture): + """Subclass of Future which represents a wait for the cancellation of a + _WaitHandleFuture using an event. + """ + + def __init__(self, ov, event, wait_handle, *, loop=None): + super().__init__(ov, event, wait_handle, loop=loop) + + self._done_callback = None + + def _schedule_callbacks(self): + super(_WaitCancelFuture, self)._schedule_callbacks() + if self._done_callback is not None: + self._done_callback(self) + + +class _WaitHandleFuture(_BaseWaitHandleFuture): + def __init__(self, ov, handle, wait_handle, proactor, *, loop=None): + super().__init__(ov, handle, wait_handle, loop=loop) + self._proactor = proactor + self._unregister_proactor = True + self._event = _overlapped.CreateEvent(None, True, False, None) + self._event_fut = None + + def _unregister_wait_cb(self, fut): + if self._event is not None: + _winapi.CloseHandle(self._event) + self._event = None + self._event_fut = None + + # If the wait was cancelled, the wait may never be signalled, so + # it's required to unregister it. Otherwise, IocpProactor.close() will + # wait forever for an event which will never come. + # + # If the IocpProactor already received the event, it's safe to call + # _unregister() because we kept a reference to the Overlapped object + # which is used as an unique key. + self._proactor._unregister(self._ov) + self._proactor = None + + super()._unregister_wait_cb(fut) + + def _unregister_wait(self): + if not self._registered: + return + self._registered = False + + try: + _overlapped.UnregisterWaitEx(self._wait_handle, self._event) + except OSError as exc: + self._wait_handle = None + if exc.winerror == _overlapped.ERROR_IO_PENDING: + # ERROR_IO_PENDING is not an error, the wait was unregistered + self._unregister_wait_cb(None) + elif exc.winerror != _overlapped.ERROR_IO_PENDING: + context = { + 'message': 'Failed to unregister the wait handle', + 'exception': exc, + 'future': self, + } + if self._source_traceback: + context['source_traceback'] = self._source_traceback + self._loop.call_exception_handler(context) + else: + self._wait_handle = None + self._event_fut = self._proactor._wait_cancel( + self._event, + self._unregister_wait_cb) class PipeServer(object): @@ -291,6 +370,7 @@ _overlapped.INVALID_HANDLE_VALUE, NULL, 0, concurrency) self._cache = {} self._registered = weakref.WeakSet() + self._unregistered = [] self._stopped_serving = weakref.WeakSet() def __repr__(self): @@ -438,6 +518,16 @@ Return a Future object. The result of the future is True if the wait completed, or False if the wait did not complete (on timeout). """ + return self._wait_for_handle(handle, timeout, False) + + def _wait_cancel(self, event, done_callback): + fut = self._wait_for_handle(event, None, True) + # add_done_callback() cannot be used because the wait may only complete + # in IocpProactor.close(), while the event loop is not running. + fut._done_callback = done_callback + return fut + + def _wait_for_handle(self, handle, timeout, _is_cancel): if timeout is None: ms = _winapi.INFINITE else: @@ -447,9 +537,13 @@ # We only create ov so we can use ov.address as a key for the cache. ov = _overlapped.Overlapped(NULL) - wh = _overlapped.RegisterWaitWithQueue( + wait_handle = _overlapped.RegisterWaitWithQueue( handle, self._iocp, ov.address, ms) - f = _WaitHandleFuture(self._iocp, ov, handle, wh, loop=self._loop) + if _is_cancel: + f = _WaitCancelFuture(ov, handle, wait_handle, loop=self._loop) + else: + f = _WaitHandleFuture(ov, handle, wait_handle, self, + loop=self._loop) if f._source_traceback: del f._source_traceback[-1] @@ -462,14 +556,6 @@ # False even though we have not timed out. return f._poll() - if f._poll(): - try: - result = f._poll() - except OSError as exc: - f.set_exception(exc) - else: - f.set_result(result) - self._cache[ov.address] = (f, ov, 0, finish_wait_for_handle) return f @@ -521,6 +607,15 @@ self._cache[ov.address] = (f, ov, obj, callback) return f + def _unregister(self, ov): + """Unregister an overlapped object. + + Call this method when its future has been cancelled. The event can + already be signalled (pending in the proactor event queue). It is also + safe if the event is never signalled (because it was cancelled). + """ + self._unregistered.append(ov) + def _get_accept_socket(self, family): s = socket.socket(family) s.settimeout(0) @@ -541,7 +636,7 @@ while True: status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms) if status is None: - return + break ms = 0 err, transferred, key, address = status @@ -576,6 +671,11 @@ f.set_result(value) self._results.append(f) + # Remove unregisted futures + for ov in self._unregistered: + self._cache.pop(ov.address, None) + self._unregistered.clear() + def _stop_serving(self, obj): # obj is a socket or pipe handle. It will be closed in # BaseProactorEventLoop._stop_serving() which will make any diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -309,6 +309,29 @@ Py_RETURN_NONE; } +PyDoc_STRVAR( + UnregisterWaitEx_doc, + "UnregisterWaitEx(WaitHandle, Event) -> None\n\n" + "Unregister wait handle.\n"); + +static PyObject * +overlapped_UnregisterWaitEx(PyObject *self, PyObject *args) +{ + HANDLE WaitHandle, Event; + BOOL ret; + + if (!PyArg_ParseTuple(args, F_HANDLE F_HANDLE, &WaitHandle, &Event)) + return NULL; + + Py_BEGIN_ALLOW_THREADS + ret = UnregisterWaitEx(WaitHandle, Event); + Py_END_ALLOW_THREADS + + if (!ret) + return SetFromWindowsErr(0); + Py_RETURN_NONE; +} + /* * Event functions -- currently only used by tests */ @@ -1319,6 +1342,8 @@ METH_VARARGS, RegisterWaitWithQueue_doc}, {"UnregisterWait", overlapped_UnregisterWait, METH_VARARGS, UnregisterWait_doc}, + {"UnregisterWaitEx", overlapped_UnregisterWaitEx, + METH_VARARGS, UnregisterWaitEx_doc}, {"CreateEvent", overlapped_CreateEvent, METH_VARARGS, CreateEvent_doc}, {"SetEvent", overlapped_SetEvent, -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 21 23:41:53 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 21 Jan 2015 22:41:53 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150121224133.69918.77209@psf.io> https://hg.python.org/cpython/rev/15671d3aaca3 changeset: 94232:15671d3aaca3 parent: 94227:0893b9ee44ea parent: 94231:fb8a093db8b1 user: Victor Stinner date: Wed Jan 21 23:40:19 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 2 +- Lib/asyncio/proactor_events.py | 12 +- Lib/asyncio/selector_events.py | 1 - Lib/asyncio/windows_events.py | 168 +++++++-- Lib/test/test_asyncio/test_proactor_events.py | 5 +- Lib/test/test_asyncio/test_streams.py | 10 +- Modules/overlapped.c | 25 + 7 files changed, 173 insertions(+), 50 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -636,7 +636,7 @@ try: yield from waiter - except Exception as exc: + except Exception: transport.close() raise diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -463,9 +463,15 @@ if f is not None: f.result() # may raise f = self._proactor.recv(self._ssock, 4096) - except: - self.close() - raise + except futures.CancelledError: + # _close_self_pipe() has been called, stop waiting for data + return + except Exception as exc: + self.call_exception_handler({ + 'message': 'Error on reading from the event loop self pipe', + 'exception': exc, + 'loop': self, + }) else: self._self_reading_future = f f.add_done_callback(self._loop_self_reading) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -10,7 +10,6 @@ import errno import functools import socket -import sys try: import ssl except ImportError: # pragma: no cover diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -78,20 +78,23 @@ self._ov = None -class _WaitHandleFuture(futures.Future): +class _BaseWaitHandleFuture(futures.Future): """Subclass of Future which represents a wait handle.""" - def __init__(self, iocp, ov, handle, wait_handle, *, loop=None): + def __init__(self, ov, handle, wait_handle, *, loop=None): super().__init__(loop=loop) if self._source_traceback: del self._source_traceback[-1] - # iocp and ov are only used by cancel() to notify IocpProactor - # that the wait was cancelled - self._iocp = iocp + # Keep a reference to the Overlapped object to keep it alive until the + # wait is unregistered self._ov = ov self._handle = handle self._wait_handle = wait_handle + # Should we call UnregisterWaitEx() if the wait completes + # or is cancelled? + self._registered = True + def _poll(self): # non-blocking wait: use a timeout of 0 millisecond return (_winapi.WaitForSingleObject(self._handle, 0) == @@ -99,21 +102,32 @@ def _repr_info(self): info = super()._repr_info() - info.insert(1, 'handle=%#x' % self._handle) - if self._wait_handle: + info.append('handle=%#x' % self._handle) + if self._handle is not None: state = 'signaled' if self._poll() else 'waiting' - info.insert(1, 'wait_handle=<%s, %#x>' - % (state, self._wait_handle)) + info.append(state) + if self._wait_handle is not None: + info.append('wait_handle=%#x' % self._wait_handle) return info + def _unregister_wait_cb(self, fut): + # The wait was unregistered: it's not safe to destroy the Overlapped + # object + self._ov = None + def _unregister_wait(self): - if self._wait_handle is None: + if not self._registered: return + self._registered = False + try: _overlapped.UnregisterWait(self._wait_handle) except OSError as exc: - # ERROR_IO_PENDING is not an error, the wait was unregistered - if exc.winerror != _overlapped.ERROR_IO_PENDING: + self._wait_handle = None + if exc.winerror == _overlapped.ERROR_IO_PENDING: + # ERROR_IO_PENDING is not an error, the wait was unregistered + self._unregister_wait_cb(None) + elif exc.winerror != _overlapped.ERROR_IO_PENDING: context = { 'message': 'Failed to unregister the wait handle', 'exception': exc, @@ -122,26 +136,91 @@ if self._source_traceback: context['source_traceback'] = self._source_traceback self._loop.call_exception_handler(context) - self._wait_handle = None - self._iocp = None - self._ov = None + else: + self._wait_handle = None + self._unregister_wait_cb(None) def cancel(self): - result = super().cancel() - if self._ov is not None: - # signal the cancellation to the overlapped object - _overlapped.PostQueuedCompletionStatus(self._iocp, True, - 0, self._ov.address) self._unregister_wait() - return result + return super().cancel() def set_exception(self, exception): + self._unregister_wait() super().set_exception(exception) - self._unregister_wait() def set_result(self, result): + self._unregister_wait() super().set_result(result) - self._unregister_wait() + + +class _WaitCancelFuture(_BaseWaitHandleFuture): + """Subclass of Future which represents a wait for the cancellation of a + _WaitHandleFuture using an event. + """ + + def __init__(self, ov, event, wait_handle, *, loop=None): + super().__init__(ov, event, wait_handle, loop=loop) + + self._done_callback = None + + def _schedule_callbacks(self): + super(_WaitCancelFuture, self)._schedule_callbacks() + if self._done_callback is not None: + self._done_callback(self) + + +class _WaitHandleFuture(_BaseWaitHandleFuture): + def __init__(self, ov, handle, wait_handle, proactor, *, loop=None): + super().__init__(ov, handle, wait_handle, loop=loop) + self._proactor = proactor + self._unregister_proactor = True + self._event = _overlapped.CreateEvent(None, True, False, None) + self._event_fut = None + + def _unregister_wait_cb(self, fut): + if self._event is not None: + _winapi.CloseHandle(self._event) + self._event = None + self._event_fut = None + + # If the wait was cancelled, the wait may never be signalled, so + # it's required to unregister it. Otherwise, IocpProactor.close() will + # wait forever for an event which will never come. + # + # If the IocpProactor already received the event, it's safe to call + # _unregister() because we kept a reference to the Overlapped object + # which is used as an unique key. + self._proactor._unregister(self._ov) + self._proactor = None + + super()._unregister_wait_cb(fut) + + def _unregister_wait(self): + if not self._registered: + return + self._registered = False + + try: + _overlapped.UnregisterWaitEx(self._wait_handle, self._event) + except OSError as exc: + self._wait_handle = None + if exc.winerror == _overlapped.ERROR_IO_PENDING: + # ERROR_IO_PENDING is not an error, the wait was unregistered + self._unregister_wait_cb(None) + elif exc.winerror != _overlapped.ERROR_IO_PENDING: + context = { + 'message': 'Failed to unregister the wait handle', + 'exception': exc, + 'future': self, + } + if self._source_traceback: + context['source_traceback'] = self._source_traceback + self._loop.call_exception_handler(context) + else: + self._wait_handle = None + self._event_fut = self._proactor._wait_cancel( + self._event, + self._unregister_wait_cb) class PipeServer(object): @@ -291,6 +370,7 @@ _overlapped.INVALID_HANDLE_VALUE, NULL, 0, concurrency) self._cache = {} self._registered = weakref.WeakSet() + self._unregistered = [] self._stopped_serving = weakref.WeakSet() def __repr__(self): @@ -438,6 +518,16 @@ Return a Future object. The result of the future is True if the wait completed, or False if the wait did not complete (on timeout). """ + return self._wait_for_handle(handle, timeout, False) + + def _wait_cancel(self, event, done_callback): + fut = self._wait_for_handle(event, None, True) + # add_done_callback() cannot be used because the wait may only complete + # in IocpProactor.close(), while the event loop is not running. + fut._done_callback = done_callback + return fut + + def _wait_for_handle(self, handle, timeout, _is_cancel): if timeout is None: ms = _winapi.INFINITE else: @@ -447,9 +537,13 @@ # We only create ov so we can use ov.address as a key for the cache. ov = _overlapped.Overlapped(NULL) - wh = _overlapped.RegisterWaitWithQueue( + wait_handle = _overlapped.RegisterWaitWithQueue( handle, self._iocp, ov.address, ms) - f = _WaitHandleFuture(self._iocp, ov, handle, wh, loop=self._loop) + if _is_cancel: + f = _WaitCancelFuture(ov, handle, wait_handle, loop=self._loop) + else: + f = _WaitHandleFuture(ov, handle, wait_handle, self, + loop=self._loop) if f._source_traceback: del f._source_traceback[-1] @@ -462,14 +556,6 @@ # False even though we have not timed out. return f._poll() - if f._poll(): - try: - result = f._poll() - except OSError as exc: - f.set_exception(exc) - else: - f.set_result(result) - self._cache[ov.address] = (f, ov, 0, finish_wait_for_handle) return f @@ -521,6 +607,15 @@ self._cache[ov.address] = (f, ov, obj, callback) return f + def _unregister(self, ov): + """Unregister an overlapped object. + + Call this method when its future has been cancelled. The event can + already be signalled (pending in the proactor event queue). It is also + safe if the event is never signalled (because it was cancelled). + """ + self._unregistered.append(ov) + def _get_accept_socket(self, family): s = socket.socket(family) s.settimeout(0) @@ -541,7 +636,7 @@ while True: status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms) if status is None: - return + break ms = 0 err, transferred, key, address = status @@ -576,6 +671,11 @@ f.set_result(value) self._results.append(f) + # Remove unregisted futures + for ov in self._unregistered: + self._cache.pop(ov.address, None) + self._unregistered.clear() + def _stop_serving(self, obj): # obj is a socket or pipe handle. It will be closed in # BaseProactorEventLoop._stop_serving() which will make any diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -523,9 +523,10 @@ def test_loop_self_reading_exception(self): self.loop.close = mock.Mock() + self.loop.call_exception_handler = mock.Mock() self.proactor.recv.side_effect = OSError() - self.assertRaises(OSError, self.loop._loop_self_reading) - self.assertTrue(self.loop.close.called) + self.loop._loop_self_reading() + self.assertTrue(self.loop.call_exception_handler.called) def test_write_to_self(self): self.loop._write_to_self() diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -415,10 +415,6 @@ def set_err(): stream.set_exception(ValueError()) - @asyncio.coroutine - def readline(): - yield from stream.readline() - t1 = asyncio.Task(stream.readline(), loop=self.loop) t2 = asyncio.Task(set_err(), loop=self.loop) @@ -429,11 +425,7 @@ def test_exception_cancel(self): stream = asyncio.StreamReader(loop=self.loop) - @asyncio.coroutine - def read_a_line(): - yield from stream.readline() - - t = asyncio.Task(read_a_line(), loop=self.loop) + t = asyncio.Task(stream.readline(), loop=self.loop) test_utils.run_briefly(self.loop) t.cancel() test_utils.run_briefly(self.loop) diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -309,6 +309,29 @@ Py_RETURN_NONE; } +PyDoc_STRVAR( + UnregisterWaitEx_doc, + "UnregisterWaitEx(WaitHandle, Event) -> None\n\n" + "Unregister wait handle.\n"); + +static PyObject * +overlapped_UnregisterWaitEx(PyObject *self, PyObject *args) +{ + HANDLE WaitHandle, Event; + BOOL ret; + + if (!PyArg_ParseTuple(args, F_HANDLE F_HANDLE, &WaitHandle, &Event)) + return NULL; + + Py_BEGIN_ALLOW_THREADS + ret = UnregisterWaitEx(WaitHandle, Event); + Py_END_ALLOW_THREADS + + if (!ret) + return SetFromWindowsErr(0); + Py_RETURN_NONE; +} + /* * Event functions -- currently only used by tests */ @@ -1319,6 +1342,8 @@ METH_VARARGS, RegisterWaitWithQueue_doc}, {"UnregisterWait", overlapped_UnregisterWait, METH_VARARGS, UnregisterWait_doc}, + {"UnregisterWaitEx", overlapped_UnregisterWaitEx, + METH_VARARGS, UnregisterWaitEx_doc}, {"CreateEvent", overlapped_CreateEvent, METH_VARARGS, CreateEvent_doc}, {"SetEvent", overlapped_SetEvent, -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 00:19:07 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 21 Jan 2015 23:19:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDk1?= =?utf-8?q?=2C_asyncio=3A_IocpProactor=2Eclose=28=29_must_not_cancel_pendi?= =?utf-8?q?ng?= Message-ID: <20150121231854.69920.91127@psf.io> https://hg.python.org/cpython/rev/d3804307cce4 changeset: 94234:d3804307cce4 branch: 3.4 user: Victor Stinner date: Thu Jan 22 00:17:54 2015 +0100 summary: Issue #23095, asyncio: IocpProactor.close() must not cancel pending _WaitCancelFuture futures files: Lib/asyncio/windows_events.py | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -163,6 +163,9 @@ self._done_callback = None + def cancel(self): + raise RuntimeError("_WaitCancelFuture must not be cancelled") + def _schedule_callbacks(self): super(_WaitCancelFuture, self)._schedule_callbacks() if self._done_callback is not None: @@ -693,6 +696,9 @@ # FIXME: Tulip issue 196: remove this case, it should not happen elif fut.done() and not fut.cancelled(): del self._cache[address] + elif isinstance(fut, _WaitCancelFuture): + # _WaitCancelFuture must not be cancelled + pass else: try: fut.cancel() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 00:19:07 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 21 Jan 2015 23:19:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQmFz?= =?utf-8?q?eEventLoop=2E=5Fcreate=5Fconnection=5Ftransport=28=29_catchs_an?= =?utf-8?q?y_exception=2C_not?= Message-ID: <20150121231854.104144.63145@psf.io> https://hg.python.org/cpython/rev/0be88d94b0c7 changeset: 94233:0be88d94b0c7 branch: 3.4 parent: 94231:fb8a093db8b1 user: Victor Stinner date: Thu Jan 22 00:17:41 2015 +0100 summary: asyncio: BaseEventLoop._create_connection_transport() catchs any exception, not only Exception files: Lib/asyncio/base_events.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -636,7 +636,7 @@ try: yield from waiter - except Exception: + except: transport.close() raise -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 00:19:07 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 21 Jan 2015 23:19:07 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150121231854.93193.72445@psf.io> https://hg.python.org/cpython/rev/69265fcade13 changeset: 94235:69265fcade13 parent: 94232:15671d3aaca3 parent: 94234:d3804307cce4 user: Victor Stinner date: Thu Jan 22 00:18:11 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 2 +- Lib/asyncio/windows_events.py | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -636,7 +636,7 @@ try: yield from waiter - except Exception: + except: transport.close() raise diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -163,6 +163,9 @@ self._done_callback = None + def cancel(self): + raise RuntimeError("_WaitCancelFuture must not be cancelled") + def _schedule_callbacks(self): super(_WaitCancelFuture, self)._schedule_callbacks() if self._done_callback is not None: @@ -693,6 +696,9 @@ # FIXME: Tulip issue 196: remove this case, it should not happen elif fut.done() and not fut.cancelled(): del self._cache[address] + elif isinstance(fut, _WaitCancelFuture): + # _WaitCancelFuture must not be cancelled + pass else: try: fut.cancel() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 02:16:40 2015 From: python-checkins at python.org (chris.angelico) Date: Thu, 22 Jan 2015 01:16:40 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_First_public_draft_of_new_PEP?= =?utf-8?q?_485_for_an_is=5Fclose=5Fto=28=29_function?= Message-ID: <20150122011626.69928.13115@psf.io> https://hg.python.org/peps/rev/a02caf3b7537 changeset: 5677:a02caf3b7537 user: Chris Angelico date: Thu Jan 22 12:15:51 2015 +1100 summary: First public draft of new PEP 485 for an is_close_to() function files: pep-0485.txt | 311 +++++++++++++++++++++++++++++++++++++++ 1 files changed, 311 insertions(+), 0 deletions(-) diff --git a/pep-0485.txt b/pep-0485.txt new file mode 100644 --- /dev/null +++ b/pep-0485.txt @@ -0,0 +1,311 @@ +PEP: 485 +Title: A Function for testing approximate equality +Version: $Revision$ +Last-Modified: $Date$ +Author: Christopher Barker +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 20-Jan-2015 +Python-Version: 3.5 +Post-History: + + +Abstract +======== + +This PEP proposes the addition of a function to the standard library +that determines whether one value is approximately equal or "close" +to another value. + + +Rationale +========= + +Floating point values contain limited precision, which results in +their being unable to exactly represent some values, and for error to +accumulate with repeated computation. As a result, it is common +advice to only use an equality comparison in very specific situations. +Often a inequality comparison fits the bill, but there are times +(often in testing) where the programmer wants to determine whether a +computed value is "close" to an expected value, without requiring them +to be exactly equal. This is common enough, particularly in testing, +and not always obvious how to do it, so it would be useful addition to +the standard library. + + +Existing Implementations +------------------------ + +The standard library includes the +``unittest.TestCase.assertAlmostEqual`` method, but it: + +* Is buried in the unittest.TestCase class + +* Is an assertion, so you can't use it as a general test (easily) + +* Uses number of decimal digits or an absolute delta, which are + particular use cases that don't provide a general relative error. + +The numpy package has the ``allclose()`` and ``isclose()`` functions. + +The statistics package tests include an implementation, used for its +unit tests. + +One can also find discussion and sample implementations on Stack +Overflow, and other help sites. + +These existing implementations indicate that this is a common need, +and not trivial to write oneself, making it a candidate for the +standard library. + + +Proposed Implementation +======================= + +NOTE: this PEP is the result of an extended discussion on the +python-ideas list [1]_. + +The new function will have the following signature:: + + is_close_to(actual, expected, tol=1e-8, abs_tol=0.0) + +``actual``: is the value that has been computed, measured, etc. + +``expected``: is the "known" value. + +``tol``: is the relative tolerance -- it is the amount of error +allowed, relative to the magnitude of the expected value. + +``abs_tol``: is an minimum absolute tolerance level -- useful for +comparisons near zero. + +Modulo error checking, etc, the function will return the result of:: + + abs(expected-actual) <= max(tol*actual, abs_tol) + + +Handling of non-finite numbers +------------------------------ + +The IEEE 754 special values of NaN, inf, and -inf will be handled +according to IEEE rules. Specifically, NaN is not considered close to +any other value, including NaN. inf and -inf are only considered close +to themselves. + + +Non-float types +--------------- + +The primary use-case is expected to be floating point numbers. +However, users may want to compare other numeric types similarly. In +theory, it should work for any type that supports ``abs()``, +comparisons, and subtraction. The code will be written and tested to +accommodate these types: + + * ``Decimal`` + + * ``int`` + + * ``Fraction`` + + * ``complex``: for complex, ``abs(z)`` will be used for scaling and + comparison. + + +Behavior near zero +------------------ + +Relative comparison is problematic if either value is zero. In this +case, the difference is relative to zero, and thus will always be +smaller than the prescribed tolerance. To handle this case, an +optional parameter, ``abs_tol`` (default 0.0) can be used to set a +minimum tolerance to be used in the case of very small relative +tolerance. That is, the values will be considered close if:: + + abs(a-b) <= abs(tol*actual) or abs(a-b) <= abs_tol + +If the user sets the rel_tol parameter to 0.0, then only the absolute +tolerance will effect the result, so this function provides an +absolute tolerance check as well. + + +Relative Difference +=================== + +There are essentially two ways to think about how close two numbers +are to each-other: absolute difference: simple ``abs(a-b)``, and +relative difference: ``abs(a-b)/scale_factor`` [2]_. The absolute +difference is trivial enough that this proposal focuses on the +relative difference. + +Usually, the scale factor is some function of the values under +consideration, for instance: + + 1) The absolute value of one of the input values + + 2) The maximum absolute value of the two + + 3) The minimum absolute value of the two. + + 4) The arithmetic mean of the two + + +Symmetry +-------- + +A relative comparison can be either symmetric or non-symmetric. For a +symmetric algorithm: + +``is_close_to(a,b)`` is always equal to ``is_close_to(b,a)`` + +This is an appealing consistency -- it mirrors the symmetry of +equality, and is less likely to confuse people. However, often the +question at hand is: + +"Is this computed or measured value within some tolerance of a known +value?" + +In this case, the user wants the relative tolerance to be specifically +scaled against the known value. It is also easier for the user to +reason about. + +This proposal uses this asymmetric test to allow this specific +definition of relative tolerance. + +Example: + +For the question: "Is the value of a within x% of b?", Using b to +scale the percent error clearly defines the result. + +However, as this approach is not symmetric, a may be within 10% of b, +but b is not within x% of a. Consider the case:: + + a = 9.0 + b = 10.0 + +The difference between a and b is 1.0. 10% of a is 0.9, so b is not +within 10% of a. But 10% of b is 1.0, so a is within 10% of b. + +Casual users might reasonably expect that if a is close to b, then b +would also be close to a. However, in the common cases, the tolerance +is quite small and often poorly defined, i.e. 1e-8, defined to only +one significant figure, so the result will be very similar regardless +of the order of the values. And if the user does care about the +precise result, s/he can take care to always pass in the two +parameters in sorted order. + +This proposed implementation uses asymmetric criteria with the scaling +value clearly identified. + + +Expected Uses +============= + +The primary expected use case is various forms of testing -- "are the +results computed near what I expect as a result?" This sort of test +may or may not be part of a formal unit testing suite. + +The function might be used also to determine if a measured value is +within an expected value. + + +Inappropriate uses +------------------ + +One use case for floating point comparison is testing the accuracy of +a numerical algorithm. However, in this case, the numerical analyst +ideally would be doing careful error propagation analysis, and should +understand exactly what to test for. It is also likely that ULP (Unit +in the Last Place) comparison may be called for. While this function +may prove useful in such situations, It is not intended to be used in +that way. + + +Other Approaches +================ + +``unittest.TestCase.assertAlmostEqual`` +--------------------------------------- + +(https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertAlmostEqual) + +Tests that values are approximately (or not approximately) equal by +computing the difference, rounding to the given number of decimal +places (default 7), and comparing to zero. + +This method was not selected for this proposal, as the use of decimal +digits is a specific, not generally useful or flexible test. + +numpy ``is_close()`` +-------------------- + +http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.isclose.html + +The numpy package provides the vectorized functions is_close() and +all_close, for similar use cases as this proposal: + +``isclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False)`` + + Returns a boolean array where two arrays are element-wise equal + within a tolerance. + + The tolerance values are positive, typically very small numbers. + The relative difference (rtol * abs(b)) and the absolute + difference atol are added together to compare against the + absolute difference between a and b + +In this approach, the absolute and relative tolerance are added +together, rather than the ``or`` method used in this proposal. This is +computationally more simple, and if relative tolerance is larger than +the absolute tolerance, then the addition will have no effect. But if +the absolute and relative tolerances are of similar magnitude, then +the allowed difference will be about twice as large as expected. + +Also, if the value passed in are small compared to the absolute +tolerance, then the relative tolerance will be completely swamped, +perhaps unexpectedly. + +This is why, in this proposal, the absolute tolerance defaults to zero +-- the user will be required to choose a value appropriate for the +values at hand. + + +Boost floating-point comparison +------------------------------- + +The Boost project ( [3]_ ) provides a floating point comparison +function. Is is a symetric approach, with both "weak" (larger of the +two relative errors) and "strong" (smaller of the two relative errors) +options. + +It was decided that a method that clearly defined which value was used +to scale the relative error would be more appropriate for the standard +library. + +References +========== + +.. [1] Python-ideas list discussion thread + (https://mail.python.org/pipermail/python-ideas/2015-January/030947.html) + +.. [2] Wikipedaia page on relative difference + (http://en.wikipedia.org/wiki/Relative_change_and_difference) + +.. [3] Boost project floating-point comparison algorithms + (http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html) + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Thu Jan 22 02:26:22 2015 From: python-checkins at python.org (chris.angelico) Date: Thu, 22 Jan 2015 01:26:22 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Minor_formatting_fixes_to_PEP?= =?utf-8?q?_485=2C_missed_in_the_first_pass?= Message-ID: <20150122012534.93173.30741@psf.io> https://hg.python.org/peps/rev/8a43472e1c12 changeset: 5678:8a43472e1c12 user: Chris Angelico date: Thu Jan 22 12:25:26 2015 +1100 summary: Minor formatting fixes to PEP 485, missed in the first pass files: pep-0485.txt | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-) diff --git a/pep-0485.txt b/pep-0485.txt --- a/pep-0485.txt +++ b/pep-0485.txt @@ -103,14 +103,14 @@ comparisons, and subtraction. The code will be written and tested to accommodate these types: - * ``Decimal`` +* ``Decimal`` - * ``int`` +* ``int`` - * ``Fraction`` +* ``Fraction`` - * ``complex``: for complex, ``abs(z)`` will be used for scaling and - comparison. +* ``complex``: for complex, ``abs(z)`` will be used for scaling and + comparison. Behavior near zero @@ -134,7 +134,7 @@ =================== There are essentially two ways to think about how close two numbers -are to each-other: absolute difference: simple ``abs(a-b)``, and +are to each-other: absolute difference: simply ``abs(a-b)``, and relative difference: ``abs(a-b)/scale_factor`` [2]_. The absolute difference is trivial enough that this proposal focuses on the relative difference. @@ -142,13 +142,13 @@ Usually, the scale factor is some function of the values under consideration, for instance: - 1) The absolute value of one of the input values +1) The absolute value of one of the input values - 2) The maximum absolute value of the two +2) The maximum absolute value of the two - 3) The minimum absolute value of the two. +3) The minimum absolute value of the two. - 4) The arithmetic mean of the two +4) The arithmetic mean of the two Symmetry -- Repository URL: https://hg.python.org/peps From solipsis at pitrou.net Thu Jan 22 08:42:07 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 22 Jan 2015 08:42:07 +0100 Subject: [Python-checkins] Daily reference leaks (69265fcade13): sum=3 Message-ID: results for 69265fcade13 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog8ypVGn', '-x'] From python-checkins at python.org Thu Jan 22 09:09:36 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 08:09:36 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40?= Message-ID: <20150122080931.104130.93995@psf.io> https://hg.python.org/cpython/rev/208f401fbbbe changeset: 94237:208f401fbbbe parent: 94235:69265fcade13 parent: 94236:d3a27a27e008 user: Victor Stinner date: Thu Jan 22 09:09:24 2015 +0100 summary: Merge 3.4 files: Lib/test/test_selectors.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -323,6 +323,8 @@ @unittest.skipIf(sys.platform == 'win32', 'select.select() cannot be used with empty fd sets') def test_empty_select(self): + # Issue #23009: Make sure EpollSelector.select() works when no FD is + # registered. s = self.SELECTOR() self.addCleanup(s.close) self.assertEqual(s.select(timeout=0), []) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 09:09:36 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 08:09:36 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDA5?= =?utf-8?q?=3A_Skip_test=5Fselectors=2Etest=5Fempty=5Fselect=28=29_on_Wind?= =?utf-8?q?ows?= Message-ID: <20150122080931.103315.12021@psf.io> https://hg.python.org/cpython/rev/d3a27a27e008 changeset: 94236:d3a27a27e008 branch: 3.4 parent: 94234:d3804307cce4 user: Victor Stinner date: Thu Jan 22 09:07:36 2015 +0100 summary: Issue #23009: Skip test_selectors.test_empty_select() on Windows files: Lib/test/test_selectors.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -322,7 +322,11 @@ self.assertEqual(bufs, [MSG] * NUM_SOCKETS) + @unittest.skipIf(sys.platform == 'win32', + 'select.select() cannot be used with empty fd sets') def test_empty_select(self): + # Issue #23009: Make sure EpollSelector.select() works when no FD is + # registered. s = self.SELECTOR() self.addCleanup(s.close) self.assertEqual(s.select(timeout=0), []) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 09:36:19 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 08:36:19 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDA5?= =?utf-8?q?=3A_Add_missing_=22import_sys=22_in_test=5Fselectors?= Message-ID: <20150122083603.104138.66545@psf.io> https://hg.python.org/cpython/rev/4f928c70f135 changeset: 94238:4f928c70f135 branch: 3.4 parent: 94236:d3a27a27e008 user: Victor Stinner date: Thu Jan 22 09:35:23 2015 +0100 summary: Issue #23009: Add missing "import sys" in test_selectors files: Lib/test/test_selectors.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -4,6 +4,7 @@ import selectors import signal import socket +import sys from test import support from time import sleep import unittest -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 09:36:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 08:36:20 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_3=2E4?= Message-ID: <20150122083604.15453.79092@psf.io> https://hg.python.org/cpython/rev/eb48f271447c changeset: 94239:eb48f271447c parent: 94237:208f401fbbbe parent: 94238:4f928c70f135 user: Victor Stinner date: Thu Jan 22 09:35:32 2015 +0100 summary: Null merge 3.4 files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 22:58:44 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 21:58:44 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogSW9j?= =?utf-8?q?pProactor=2Eclose=28=29_doesn=27t_cancel_anymore_futures_which_?= =?utf-8?q?are_already?= Message-ID: <20150122215832.118222.74992@psf.io> https://hg.python.org/cpython/rev/7d5c23b87eae changeset: 94240:7d5c23b87eae branch: 3.4 parent: 94238:4f928c70f135 user: Victor Stinner date: Thu Jan 22 22:47:13 2015 +0100 summary: asyncio: IocpProactor.close() doesn't cancel anymore futures which are already cancelled files: Lib/asyncio/windows_events.py | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -693,12 +693,16 @@ # queues a task to Windows' thread pool. This cannot # be cancelled, so just forget it. del self._cache[address] - # FIXME: Tulip issue 196: remove this case, it should not happen - elif fut.done() and not fut.cancelled(): - del self._cache[address] + elif fut.cancelled(): + # Nothing to do with cancelled futures + pass elif isinstance(fut, _WaitCancelFuture): # _WaitCancelFuture must not be cancelled pass + elif fut.done(): + # FIXME: Tulip issue 196: remove this case, it should not + # happen + del self._cache[address] else: try: fut.cancel() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 22:58:44 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 21:58:44 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjkz?= =?utf-8?q?=2C_asyncio=3A_Rewrite_IocpProactor=2Econnect=5Fpipe=28=29?= Message-ID: <20150122215832.118224.492@psf.io> https://hg.python.org/cpython/rev/1e3a1af0705f changeset: 94241:1e3a1af0705f branch: 3.4 user: Victor Stinner date: Thu Jan 22 22:55:08 2015 +0100 summary: Issue #23293, asyncio: Rewrite IocpProactor.connect_pipe() Add _overlapped.ConnectPipe() which tries to connect to the pipe for asynchronous I/O (overlapped): call CreateFile() in a loop until it doesn't fail with ERROR_PIPE_BUSY. Use an increasing delay between 1 ms and 100 ms. Remove Overlapped.WaitNamedPipeAndConnect() which is no more used. files: Lib/asyncio/windows_events.py | 45 +++++--- Modules/overlapped.c | 115 ++++----------------- 2 files changed, 49 insertions(+), 111 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -29,6 +29,12 @@ ERROR_CONNECTION_REFUSED = 1225 ERROR_CONNECTION_ABORTED = 1236 +# Initial delay in seconds for connect_pipe() before retrying to connect +CONNECT_PIPE_INIT_DELAY = 0.001 + +# Maximum delay in seconds for connect_pipe() before retrying to connect +CONNECT_PIPE_MAX_DELAY = 0.100 + class _OverlappedFuture(futures.Future): """Subclass of Future which represents an overlapped operation. @@ -495,25 +501,28 @@ return self._register(ov, pipe, finish_accept_pipe, register=False) + def _connect_pipe(self, fut, address, delay): + # Unfortunately there is no way to do an overlapped connect to a pipe. + # Call CreateFile() in a loop until it doesn't fail with + # ERROR_PIPE_BUSY + try: + handle = _overlapped.ConnectPipe(address) + except OSError as exc: + if exc.winerror == _overlapped.ERROR_PIPE_BUSY: + # Polling: retry later + delay = min(delay * 2, CONNECT_PIPE_MAX_DELAY) + self._loop.call_later(delay, + self._connect_pipe, fut, address, delay) + else: + fut.set_exception(exc) + else: + pipe = windows_utils.PipeHandle(handle) + fut.set_result(pipe) + def connect_pipe(self, address): - ov = _overlapped.Overlapped(NULL) - ov.WaitNamedPipeAndConnect(address, self._iocp, ov.address) - - def finish_connect_pipe(err, handle, ov): - # err, handle were arguments passed to PostQueuedCompletionStatus() - # in a function run in a thread pool. - if err == _overlapped.ERROR_SEM_TIMEOUT: - # Connection did not succeed within time limit. - msg = _overlapped.FormatMessage(err) - raise ConnectionRefusedError(0, msg, None, err) - elif err != 0: - msg = _overlapped.FormatMessage(err) - raise OSError(0, msg, None, err) - else: - return windows_utils.PipeHandle(handle) - - return self._register(ov, None, finish_connect_pipe, - wait_for_post=True) + fut = futures.Future(loop=self._loop) + self._connect_pipe(fut, address, CONNECT_PIPE_INIT_DELAY) + return fut def wait_for_handle(self, handle, timeout=None): """Wait for a handle. diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -52,12 +52,6 @@ }; } OverlappedObject; -typedef struct { - OVERLAPPED *Overlapped; - HANDLE IocpHandle; - char Address[1]; -} WaitNamedPipeAndConnectContext; - /* * Map Windows error codes to subclasses of OSError */ @@ -1133,99 +1127,33 @@ } } -/* Unfortunately there is no way to do an overlapped connect to a - pipe. We instead use WaitNamedPipe() and CreateFile() in a thread - pool thread. If a connection succeeds within a time limit (10 - seconds) then PostQueuedCompletionStatus() is used to return the - pipe handle to the completion port. */ - -static DWORD WINAPI -WaitNamedPipeAndConnectInThread(WaitNamedPipeAndConnectContext *ctx) -{ - HANDLE PipeHandle = INVALID_HANDLE_VALUE; - DWORD Start = GetTickCount(); - DWORD Deadline = Start + 10*1000; - DWORD Error = 0; - DWORD Timeout; - BOOL Success; - - for ( ; ; ) { - Timeout = Deadline - GetTickCount(); - if ((int)Timeout < 0) - break; - Success = WaitNamedPipe(ctx->Address, Timeout); - Error = Success ? ERROR_SUCCESS : GetLastError(); - switch (Error) { - case ERROR_SUCCESS: - PipeHandle = CreateFile(ctx->Address, - GENERIC_READ | GENERIC_WRITE, - 0, NULL, OPEN_EXISTING, - FILE_FLAG_OVERLAPPED, NULL); - if (PipeHandle == INVALID_HANDLE_VALUE) - continue; - break; - case ERROR_SEM_TIMEOUT: - continue; - } - break; - } - if (!PostQueuedCompletionStatus(ctx->IocpHandle, Error, - (ULONG_PTR)PipeHandle, ctx->Overlapped)) - CloseHandle(PipeHandle); - free(ctx); - return 0; -} - PyDoc_STRVAR( - Overlapped_WaitNamedPipeAndConnect_doc, - "WaitNamedPipeAndConnect(addr, iocp_handle) -> Overlapped[pipe_handle]\n\n" - "Start overlapped connection to address, notifying iocp_handle when\n" - "finished"); + ConnectPipe_doc, + "ConnectPipe(addr) -> pipe_handle\n\n" + "Connect to the pipe for asynchronous I/O (overlapped)."); static PyObject * -Overlapped_WaitNamedPipeAndConnect(OverlappedObject *self, PyObject *args) +ConnectPipe(OverlappedObject *self, PyObject *args) { - char *Address; - Py_ssize_t AddressLength; - HANDLE IocpHandle; - OVERLAPPED Overlapped; - BOOL ret; - DWORD err; - WaitNamedPipeAndConnectContext *ctx; - Py_ssize_t ContextLength; + PyObject *AddressObj; + wchar_t *Address; + HANDLE PipeHandle; - if (!PyArg_ParseTuple(args, "s#" F_HANDLE F_POINTER, - &Address, &AddressLength, &IocpHandle, &Overlapped)) + if (!PyArg_ParseTuple(args, "U", &AddressObj)) return NULL; - if (self->type != TYPE_NONE) { - PyErr_SetString(PyExc_ValueError, "operation already attempted"); + Address = PyUnicode_AsWideCharString(AddressObj, NULL); + if (Address == NULL) return NULL; - } - ContextLength = (AddressLength + - offsetof(WaitNamedPipeAndConnectContext, Address)); - ctx = calloc(1, ContextLength + 1); - if (ctx == NULL) - return PyErr_NoMemory(); - memcpy(ctx->Address, Address, AddressLength + 1); - ctx->Overlapped = &self->overlapped; - ctx->IocpHandle = IocpHandle; - - self->type = TYPE_WAIT_NAMED_PIPE_AND_CONNECT; - self->handle = NULL; - - Py_BEGIN_ALLOW_THREADS - ret = QueueUserWorkItem(WaitNamedPipeAndConnectInThread, ctx, - WT_EXECUTELONGFUNCTION); - Py_END_ALLOW_THREADS - - mark_as_completed(&self->overlapped); - - self->error = err = ret ? ERROR_SUCCESS : GetLastError(); - if (!ret) - return SetFromWindowsErr(err); - Py_RETURN_NONE; + PipeHandle = CreateFileW(Address, + GENERIC_READ | GENERIC_WRITE, + 0, NULL, OPEN_EXISTING, + FILE_FLAG_OVERLAPPED, NULL); + PyMem_Free(Address); + if (PipeHandle == INVALID_HANDLE_VALUE) + return SetFromWindowsErr(0); + return Py_BuildValue(F_HANDLE, PipeHandle); } static PyObject* @@ -1262,9 +1190,6 @@ METH_VARARGS, Overlapped_DisconnectEx_doc}, {"ConnectNamedPipe", (PyCFunction) Overlapped_ConnectNamedPipe, METH_VARARGS, Overlapped_ConnectNamedPipe_doc}, - {"WaitNamedPipeAndConnect", - (PyCFunction) Overlapped_WaitNamedPipeAndConnect, - METH_VARARGS, Overlapped_WaitNamedPipeAndConnect_doc}, {NULL} }; @@ -1350,6 +1275,9 @@ METH_VARARGS, SetEvent_doc}, {"ResetEvent", overlapped_ResetEvent, METH_VARARGS, ResetEvent_doc}, + {"ConnectPipe", + (PyCFunction) ConnectPipe, + METH_VARARGS, ConnectPipe_doc}, {NULL} }; @@ -1394,6 +1322,7 @@ WINAPI_CONSTANT(F_DWORD, ERROR_IO_PENDING); WINAPI_CONSTANT(F_DWORD, ERROR_NETNAME_DELETED); WINAPI_CONSTANT(F_DWORD, ERROR_SEM_TIMEOUT); + WINAPI_CONSTANT(F_DWORD, ERROR_PIPE_BUSY); WINAPI_CONSTANT(F_DWORD, INFINITE); WINAPI_CONSTANT(F_HANDLE, INVALID_HANDLE_VALUE); WINAPI_CONSTANT(F_HANDLE, NULL); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 22:58:44 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 21:58:44 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150122215833.87111.55708@psf.io> https://hg.python.org/cpython/rev/1c1c0f40a64b changeset: 94242:1c1c0f40a64b parent: 94239:eb48f271447c parent: 94241:1e3a1af0705f user: Victor Stinner date: Thu Jan 22 22:55:31 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/windows_events.py | 55 ++++++---- Modules/overlapped.c | 115 ++++----------------- 2 files changed, 56 insertions(+), 114 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -29,6 +29,12 @@ ERROR_CONNECTION_REFUSED = 1225 ERROR_CONNECTION_ABORTED = 1236 +# Initial delay in seconds for connect_pipe() before retrying to connect +CONNECT_PIPE_INIT_DELAY = 0.001 + +# Maximum delay in seconds for connect_pipe() before retrying to connect +CONNECT_PIPE_MAX_DELAY = 0.100 + class _OverlappedFuture(futures.Future): """Subclass of Future which represents an overlapped operation. @@ -495,25 +501,28 @@ return self._register(ov, pipe, finish_accept_pipe, register=False) + def _connect_pipe(self, fut, address, delay): + # Unfortunately there is no way to do an overlapped connect to a pipe. + # Call CreateFile() in a loop until it doesn't fail with + # ERROR_PIPE_BUSY + try: + handle = _overlapped.ConnectPipe(address) + except OSError as exc: + if exc.winerror == _overlapped.ERROR_PIPE_BUSY: + # Polling: retry later + delay = min(delay * 2, CONNECT_PIPE_MAX_DELAY) + self._loop.call_later(delay, + self._connect_pipe, fut, address, delay) + else: + fut.set_exception(exc) + else: + pipe = windows_utils.PipeHandle(handle) + fut.set_result(pipe) + def connect_pipe(self, address): - ov = _overlapped.Overlapped(NULL) - ov.WaitNamedPipeAndConnect(address, self._iocp, ov.address) - - def finish_connect_pipe(err, handle, ov): - # err, handle were arguments passed to PostQueuedCompletionStatus() - # in a function run in a thread pool. - if err == _overlapped.ERROR_SEM_TIMEOUT: - # Connection did not succeed within time limit. - msg = _overlapped.FormatMessage(err) - raise ConnectionRefusedError(0, msg, None, err) - elif err != 0: - msg = _overlapped.FormatMessage(err) - raise OSError(0, msg, None, err) - else: - return windows_utils.PipeHandle(handle) - - return self._register(ov, None, finish_connect_pipe, - wait_for_post=True) + fut = futures.Future(loop=self._loop) + self._connect_pipe(fut, address, CONNECT_PIPE_INIT_DELAY) + return fut def wait_for_handle(self, handle, timeout=None): """Wait for a handle. @@ -693,12 +702,16 @@ # queues a task to Windows' thread pool. This cannot # be cancelled, so just forget it. del self._cache[address] - # FIXME: Tulip issue 196: remove this case, it should not happen - elif fut.done() and not fut.cancelled(): - del self._cache[address] + elif fut.cancelled(): + # Nothing to do with cancelled futures + pass elif isinstance(fut, _WaitCancelFuture): # _WaitCancelFuture must not be cancelled pass + elif fut.done(): + # FIXME: Tulip issue 196: remove this case, it should not + # happen + del self._cache[address] else: try: fut.cancel() diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -52,12 +52,6 @@ }; } OverlappedObject; -typedef struct { - OVERLAPPED *Overlapped; - HANDLE IocpHandle; - char Address[1]; -} WaitNamedPipeAndConnectContext; - /* * Map Windows error codes to subclasses of OSError */ @@ -1133,99 +1127,33 @@ } } -/* Unfortunately there is no way to do an overlapped connect to a - pipe. We instead use WaitNamedPipe() and CreateFile() in a thread - pool thread. If a connection succeeds within a time limit (10 - seconds) then PostQueuedCompletionStatus() is used to return the - pipe handle to the completion port. */ - -static DWORD WINAPI -WaitNamedPipeAndConnectInThread(WaitNamedPipeAndConnectContext *ctx) -{ - HANDLE PipeHandle = INVALID_HANDLE_VALUE; - DWORD Start = GetTickCount(); - DWORD Deadline = Start + 10*1000; - DWORD Error = 0; - DWORD Timeout; - BOOL Success; - - for ( ; ; ) { - Timeout = Deadline - GetTickCount(); - if ((int)Timeout < 0) - break; - Success = WaitNamedPipe(ctx->Address, Timeout); - Error = Success ? ERROR_SUCCESS : GetLastError(); - switch (Error) { - case ERROR_SUCCESS: - PipeHandle = CreateFile(ctx->Address, - GENERIC_READ | GENERIC_WRITE, - 0, NULL, OPEN_EXISTING, - FILE_FLAG_OVERLAPPED, NULL); - if (PipeHandle == INVALID_HANDLE_VALUE) - continue; - break; - case ERROR_SEM_TIMEOUT: - continue; - } - break; - } - if (!PostQueuedCompletionStatus(ctx->IocpHandle, Error, - (ULONG_PTR)PipeHandle, ctx->Overlapped)) - CloseHandle(PipeHandle); - free(ctx); - return 0; -} - PyDoc_STRVAR( - Overlapped_WaitNamedPipeAndConnect_doc, - "WaitNamedPipeAndConnect(addr, iocp_handle) -> Overlapped[pipe_handle]\n\n" - "Start overlapped connection to address, notifying iocp_handle when\n" - "finished"); + ConnectPipe_doc, + "ConnectPipe(addr) -> pipe_handle\n\n" + "Connect to the pipe for asynchronous I/O (overlapped)."); static PyObject * -Overlapped_WaitNamedPipeAndConnect(OverlappedObject *self, PyObject *args) +ConnectPipe(OverlappedObject *self, PyObject *args) { - char *Address; - Py_ssize_t AddressLength; - HANDLE IocpHandle; - OVERLAPPED Overlapped; - BOOL ret; - DWORD err; - WaitNamedPipeAndConnectContext *ctx; - Py_ssize_t ContextLength; + PyObject *AddressObj; + wchar_t *Address; + HANDLE PipeHandle; - if (!PyArg_ParseTuple(args, "s#" F_HANDLE F_POINTER, - &Address, &AddressLength, &IocpHandle, &Overlapped)) + if (!PyArg_ParseTuple(args, "U", &AddressObj)) return NULL; - if (self->type != TYPE_NONE) { - PyErr_SetString(PyExc_ValueError, "operation already attempted"); + Address = PyUnicode_AsWideCharString(AddressObj, NULL); + if (Address == NULL) return NULL; - } - ContextLength = (AddressLength + - offsetof(WaitNamedPipeAndConnectContext, Address)); - ctx = calloc(1, ContextLength + 1); - if (ctx == NULL) - return PyErr_NoMemory(); - memcpy(ctx->Address, Address, AddressLength + 1); - ctx->Overlapped = &self->overlapped; - ctx->IocpHandle = IocpHandle; - - self->type = TYPE_WAIT_NAMED_PIPE_AND_CONNECT; - self->handle = NULL; - - Py_BEGIN_ALLOW_THREADS - ret = QueueUserWorkItem(WaitNamedPipeAndConnectInThread, ctx, - WT_EXECUTELONGFUNCTION); - Py_END_ALLOW_THREADS - - mark_as_completed(&self->overlapped); - - self->error = err = ret ? ERROR_SUCCESS : GetLastError(); - if (!ret) - return SetFromWindowsErr(err); - Py_RETURN_NONE; + PipeHandle = CreateFileW(Address, + GENERIC_READ | GENERIC_WRITE, + 0, NULL, OPEN_EXISTING, + FILE_FLAG_OVERLAPPED, NULL); + PyMem_Free(Address); + if (PipeHandle == INVALID_HANDLE_VALUE) + return SetFromWindowsErr(0); + return Py_BuildValue(F_HANDLE, PipeHandle); } static PyObject* @@ -1262,9 +1190,6 @@ METH_VARARGS, Overlapped_DisconnectEx_doc}, {"ConnectNamedPipe", (PyCFunction) Overlapped_ConnectNamedPipe, METH_VARARGS, Overlapped_ConnectNamedPipe_doc}, - {"WaitNamedPipeAndConnect", - (PyCFunction) Overlapped_WaitNamedPipeAndConnect, - METH_VARARGS, Overlapped_WaitNamedPipeAndConnect_doc}, {NULL} }; @@ -1350,6 +1275,9 @@ METH_VARARGS, SetEvent_doc}, {"ResetEvent", overlapped_ResetEvent, METH_VARARGS, ResetEvent_doc}, + {"ConnectPipe", + (PyCFunction) ConnectPipe, + METH_VARARGS, ConnectPipe_doc}, {NULL} }; @@ -1394,6 +1322,7 @@ WINAPI_CONSTANT(F_DWORD, ERROR_IO_PENDING); WINAPI_CONSTANT(F_DWORD, ERROR_NETNAME_DELETED); WINAPI_CONSTANT(F_DWORD, ERROR_SEM_TIMEOUT); + WINAPI_CONSTANT(F_DWORD, ERROR_PIPE_BUSY); WINAPI_CONSTANT(F_DWORD, INFINITE); WINAPI_CONSTANT(F_HANDLE, INVALID_HANDLE_VALUE); WINAPI_CONSTANT(F_HANDLE, NULL); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 23:38:55 2015 From: python-checkins at python.org (gregory.p.smith) Date: Thu, 22 Jan 2015 22:38:55 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Break_up_TestCommandLine=2Etest=5Fenv=5Fvar_into_four_di?= =?utf-8?q?stinct_tests=2E?= Message-ID: <20150122223844.84279.23009@psf.io> https://hg.python.org/cpython/rev/e94fa708c1c6 changeset: 94244:e94fa708c1c6 parent: 94242:1c1c0f40a64b parent: 94243:76e3f80eb680 user: Gregory P. Smith date: Thu Jan 22 14:38:26 2015 -0800 summary: Break up TestCommandLine.test_env_var into four distinct tests. files: Lib/test/test_tracemalloc.py | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py --- a/Lib/test/test_tracemalloc.py +++ b/Lib/test/test_tracemalloc.py @@ -748,26 +748,28 @@ class TestCommandLine(unittest.TestCase): - def test_env_var(self): + def test_env_var_disabled_by_default(self): # not tracing by default code = 'import tracemalloc; print(tracemalloc.is_tracing())' ok, stdout, stderr = assert_python_ok('-c', code) stdout = stdout.rstrip() self.assertEqual(stdout, b'False') - # PYTHON* environment variables must be ignored when -E option is - # present + def test_env_var_ignored_with_E(self): + """PYTHON* environment variables must be ignored when -E is present.""" code = 'import tracemalloc; print(tracemalloc.is_tracing())' ok, stdout, stderr = assert_python_ok('-E', '-c', code, PYTHONTRACEMALLOC='1') stdout = stdout.rstrip() self.assertEqual(stdout, b'False') + def test_env_var_enabled_at_startup(self): # tracing at startup code = 'import tracemalloc; print(tracemalloc.is_tracing())' ok, stdout, stderr = assert_python_ok('-c', code, PYTHONTRACEMALLOC='1') stdout = stdout.rstrip() self.assertEqual(stdout, b'True') + def test_env_limit(self): # start and set the number of frames code = 'import tracemalloc; print(tracemalloc.get_traceback_limit())' ok, stdout, stderr = assert_python_ok('-c', code, PYTHONTRACEMALLOC='10') -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 23:38:55 2015 From: python-checkins at python.org (gregory.p.smith) Date: Thu, 22 Jan 2015 22:38:55 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Break_up_TestC?= =?utf-8?q?ommandLine=2Etest=5Fenv=5Fvar_into_four_distinct_tests=2E?= Message-ID: <20150122223844.55106.29183@psf.io> https://hg.python.org/cpython/rev/76e3f80eb680 changeset: 94243:76e3f80eb680 branch: 3.4 parent: 94241:1e3a1af0705f user: Gregory P. Smith date: Thu Jan 22 14:38:00 2015 -0800 summary: Break up TestCommandLine.test_env_var into four distinct tests. files: Lib/test/test_tracemalloc.py | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py --- a/Lib/test/test_tracemalloc.py +++ b/Lib/test/test_tracemalloc.py @@ -748,26 +748,28 @@ class TestCommandLine(unittest.TestCase): - def test_env_var(self): + def test_env_var_disabled_by_default(self): # not tracing by default code = 'import tracemalloc; print(tracemalloc.is_tracing())' ok, stdout, stderr = assert_python_ok('-c', code) stdout = stdout.rstrip() self.assertEqual(stdout, b'False') - # PYTHON* environment variables must be ignored when -E option is - # present + def test_env_var_ignored_with_E(self): + """PYTHON* environment variables must be ignored when -E is present.""" code = 'import tracemalloc; print(tracemalloc.is_tracing())' ok, stdout, stderr = assert_python_ok('-E', '-c', code, PYTHONTRACEMALLOC='1') stdout = stdout.rstrip() self.assertEqual(stdout, b'False') + def test_env_var_enabled_at_startup(self): # tracing at startup code = 'import tracemalloc; print(tracemalloc.is_tracing())' ok, stdout, stderr = assert_python_ok('-c', code, PYTHONTRACEMALLOC='1') stdout = stdout.rstrip() self.assertEqual(stdout, b'True') + def test_env_limit(self): # start and set the number of frames code = 'import tracemalloc; print(tracemalloc.get_traceback_limit())' ok, stdout, stderr = assert_python_ok('-c', code, PYTHONTRACEMALLOC='10') -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 23:51:59 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 22:51:59 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150122225150.55120.55898@psf.io> https://hg.python.org/cpython/rev/5dc7df4e4143 changeset: 94246:5dc7df4e4143 parent: 94244:e94fa708c1c6 parent: 94245:6f0b03c17b50 user: Victor Stinner date: Thu Jan 22 23:50:34 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/windows_events.py | 41 +++++++++------------- Modules/overlapped.c | 4 +- 2 files changed, 19 insertions(+), 26 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -490,16 +490,21 @@ def accept_pipe(self, pipe): self._register_with_iocp(pipe) ov = _overlapped.Overlapped(NULL) - ov.ConnectNamedPipe(pipe.fileno()) + connected = ov.ConnectNamedPipe(pipe.fileno()) + + if connected: + # ConnectNamePipe() failed with ERROR_PIPE_CONNECTED which means + # that the pipe is connected. There is no need to wait for the + # completion of the connection. + f = futures.Future(loop=self._loop) + f.set_result(pipe) + return f def finish_accept_pipe(trans, key, ov): ov.getresult() return pipe - # FIXME: Tulip issue 196: why do we need register=False? - # See also the comment in the _register() method - return self._register(ov, pipe, finish_accept_pipe, - register=False) + return self._register(ov, pipe, finish_accept_pipe) def _connect_pipe(self, fut, address, delay): # Unfortunately there is no way to do an overlapped connect to a pipe. @@ -581,15 +586,14 @@ # to avoid sending notifications to completion port of ops # that succeed immediately. - def _register(self, ov, obj, callback, - wait_for_post=False, register=True): + def _register(self, ov, obj, callback): # Return a future which will be set with the result of the # operation when it completes. The future's value is actually # the value returned by callback(). f = _OverlappedFuture(ov, loop=self._loop) if f._source_traceback: del f._source_traceback[-1] - if not ov.pending and not wait_for_post: + if not ov.pending: # The operation has completed, so no need to postpone the # work. We cannot take this short cut if we need the # NumberOfBytes, CompletionKey values returned by @@ -605,18 +609,11 @@ # Register the overlapped operation to keep a reference to the # OVERLAPPED object, otherwise the memory is freed and Windows may # read uninitialized memory. - # - # For an unknown reason, ConnectNamedPipe() behaves differently: - # the completion is not notified by GetOverlappedResult() if we - # already called GetOverlappedResult(). For this specific case, we - # don't expect notification (register is set to False). - else: - register = True - if register: - # Register the overlapped operation for later. Note that - # we only store obj to prevent it from being garbage - # collected too early. - self._cache[ov.address] = (f, ov, obj, callback) + + # Register the overlapped operation for later. Note that + # we only store obj to prevent it from being garbage + # collected too early. + self._cache[ov.address] = (f, ov, obj, callback) return f def _unregister(self, ov): @@ -708,10 +705,6 @@ elif isinstance(fut, _WaitCancelFuture): # _WaitCancelFuture must not be cancelled pass - elif fut.done(): - # FIXME: Tulip issue 196: remove this case, it should not - # happen - del self._cache[address] else: try: fut.cancel() diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -1117,10 +1117,10 @@ switch (err) { case ERROR_PIPE_CONNECTED: mark_as_completed(&self->overlapped); - Py_RETURN_NONE; + Py_RETURN_TRUE; case ERROR_SUCCESS: case ERROR_IO_PENDING: - Py_RETURN_NONE; + Py_RETURN_FALSE; default: self->type = TYPE_NOT_STARTED; return SetFromWindowsErr(err); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 22 23:52:00 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 22 Jan 2015 22:52:00 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?= =?utf-8?q?ip_issue_204=3A_Fix_IocpProactor=2Eaccept=5Fpipe=28=29?= Message-ID: <20150122225150.55102.68777@psf.io> https://hg.python.org/cpython/rev/6f0b03c17b50 changeset: 94245:6f0b03c17b50 branch: 3.4 parent: 94243:76e3f80eb680 user: Victor Stinner date: Thu Jan 22 23:50:03 2015 +0100 summary: asyncio, Tulip issue 204: Fix IocpProactor.accept_pipe() Overlapped.ConnectNamedPipe() now returns a boolean: True if the pipe is connected (if ConnectNamedPipe() failed with ERROR_PIPE_CONNECTED), False if the connection is in progress. This change removes multiple hacks in IocpProactor. files: Lib/asyncio/windows_events.py | 41 +++++++++------------- Modules/overlapped.c | 4 +- 2 files changed, 19 insertions(+), 26 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -490,16 +490,21 @@ def accept_pipe(self, pipe): self._register_with_iocp(pipe) ov = _overlapped.Overlapped(NULL) - ov.ConnectNamedPipe(pipe.fileno()) + connected = ov.ConnectNamedPipe(pipe.fileno()) + + if connected: + # ConnectNamePipe() failed with ERROR_PIPE_CONNECTED which means + # that the pipe is connected. There is no need to wait for the + # completion of the connection. + f = futures.Future(loop=self._loop) + f.set_result(pipe) + return f def finish_accept_pipe(trans, key, ov): ov.getresult() return pipe - # FIXME: Tulip issue 196: why do we need register=False? - # See also the comment in the _register() method - return self._register(ov, pipe, finish_accept_pipe, - register=False) + return self._register(ov, pipe, finish_accept_pipe) def _connect_pipe(self, fut, address, delay): # Unfortunately there is no way to do an overlapped connect to a pipe. @@ -581,15 +586,14 @@ # to avoid sending notifications to completion port of ops # that succeed immediately. - def _register(self, ov, obj, callback, - wait_for_post=False, register=True): + def _register(self, ov, obj, callback): # Return a future which will be set with the result of the # operation when it completes. The future's value is actually # the value returned by callback(). f = _OverlappedFuture(ov, loop=self._loop) if f._source_traceback: del f._source_traceback[-1] - if not ov.pending and not wait_for_post: + if not ov.pending: # The operation has completed, so no need to postpone the # work. We cannot take this short cut if we need the # NumberOfBytes, CompletionKey values returned by @@ -605,18 +609,11 @@ # Register the overlapped operation to keep a reference to the # OVERLAPPED object, otherwise the memory is freed and Windows may # read uninitialized memory. - # - # For an unknown reason, ConnectNamedPipe() behaves differently: - # the completion is not notified by GetOverlappedResult() if we - # already called GetOverlappedResult(). For this specific case, we - # don't expect notification (register is set to False). - else: - register = True - if register: - # Register the overlapped operation for later. Note that - # we only store obj to prevent it from being garbage - # collected too early. - self._cache[ov.address] = (f, ov, obj, callback) + + # Register the overlapped operation for later. Note that + # we only store obj to prevent it from being garbage + # collected too early. + self._cache[ov.address] = (f, ov, obj, callback) return f def _unregister(self, ov): @@ -708,10 +705,6 @@ elif isinstance(fut, _WaitCancelFuture): # _WaitCancelFuture must not be cancelled pass - elif fut.done(): - # FIXME: Tulip issue 196: remove this case, it should not - # happen - del self._cache[address] else: try: fut.cancel() diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -1117,10 +1117,10 @@ switch (err) { case ERROR_PIPE_CONNECTED: mark_as_completed(&self->overlapped); - Py_RETURN_NONE; + Py_RETURN_TRUE; case ERROR_SUCCESS: case ERROR_IO_PENDING: - Py_RETURN_NONE; + Py_RETURN_FALSE; default: self->type = TYPE_NOT_STARTED; return SetFromWindowsErr(err); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 01:50:48 2015 From: python-checkins at python.org (chris.angelico) Date: Fri, 23 Jan 2015 00:50:48 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Apply_Chris_Barker=27s_latest?= =?utf-8?q?_PEP_485_changes?= Message-ID: <20150123005038.75802.55538@psf.io> https://hg.python.org/peps/rev/d234c5ff7ef6 changeset: 5679:d234c5ff7ef6 user: Chris Angelico date: Fri Jan 23 11:50:29 2015 +1100 summary: Apply Chris Barker's latest PEP 485 changes files: pep-0485.txt | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diff --git a/pep-0485.txt b/pep-0485.txt --- a/pep-0485.txt +++ b/pep-0485.txt @@ -16,7 +16,7 @@ This PEP proposes the addition of a function to the standard library that determines whether one value is approximately equal or "close" -to another value. +to another value. Rationale @@ -82,7 +82,7 @@ Modulo error checking, etc, the function will return the result of:: - abs(expected-actual) <= max(tol*actual, abs_tol) + abs(expected-actual) <= max(tol*expected, abs_tol) Handling of non-finite numbers @@ -103,7 +103,7 @@ comparisons, and subtraction. The code will be written and tested to accommodate these types: -* ``Decimal`` +* ``Decimal``: for Decimal, the tolerance must be set to a Decimal type. * ``int`` @@ -123,12 +123,16 @@ minimum tolerance to be used in the case of very small relative tolerance. That is, the values will be considered close if:: - abs(a-b) <= abs(tol*actual) or abs(a-b) <= abs_tol + abs(a-b) <= abs(tol*expected) or abs(a-b) <= abs_tol If the user sets the rel_tol parameter to 0.0, then only the absolute tolerance will effect the result, so this function provides an absolute tolerance check as well. +A sample implementation is available (as of Jan 22, 2015) on gitHub: + +https://github.com/PythonCHB/close_pep/blob/master/is_close_to.py + Relative Difference =================== -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 23 02:54:15 2015 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 23 Jan 2015 01:54:15 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Only_pass_-E_t?= =?utf-8?q?o_the_child_interpreter_if_our_interpreter_was_running_in_that?= Message-ID: <20150123015350.87111.64184@psf.io> https://hg.python.org/cpython/rev/7f3ac2ce24ed changeset: 94247:7f3ac2ce24ed branch: 3.4 parent: 94245:6f0b03c17b50 user: Gregory P. Smith date: Thu Jan 22 17:33:28 2015 -0800 summary: Only pass -E to the child interpreter if our interpreter was running in that mode. Explicitly remove the PYTHONFAULTHANDLER environment variable before launching a child interpreter when its presence would impact the test (the reason -E was being used in the first place). This enables running the test in an environment where other Python environment variables must be set in order for things to run (such as using PYTHONHOME to tell an embedded interpreter where it should think it lives). files: Lib/test/test_faulthandler.py | 22 +++++++++++++++------- 1 files changed, 15 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -250,17 +250,25 @@ def test_disabled_by_default(self): # By default, the module should be disabled code = "import faulthandler; print(faulthandler.is_enabled())" - args = (sys.executable, '-E', '-c', code) - # don't use assert_python_ok() because it always enable faulthandler - output = subprocess.check_output(args) + args = filter(None, (sys.executable, + "-E" if sys.flags.ignore_environment else "", + "-c", code)) + env = os.environ.copy() + env.pop("PYTHONFAULTHANDLER", None) + # don't use assert_python_ok() because it always enables faulthandler + output = subprocess.check_output(args, env=env) self.assertEqual(output.rstrip(), b"False") def test_sys_xoptions(self): # Test python -X faulthandler code = "import faulthandler; print(faulthandler.is_enabled())" - args = (sys.executable, "-E", "-X", "faulthandler", "-c", code) - # don't use assert_python_ok() because it always enable faulthandler - output = subprocess.check_output(args) + args = filter(None, (sys.executable, + "-E" if sys.flags.ignore_environment else "", + "-X", "faulthandler", "-c", code)) + env = os.environ.copy() + env.pop("PYTHONFAULTHANDLER", None) + # don't use assert_python_ok() because it always enables faulthandler + output = subprocess.check_output(args, env=env) self.assertEqual(output.rstrip(), b"True") def test_env_var(self): @@ -269,7 +277,7 @@ args = (sys.executable, "-c", code) env = os.environ.copy() env['PYTHONFAULTHANDLER'] = '' - # don't use assert_python_ok() because it always enable faulthandler + # don't use assert_python_ok() because it always enables faulthandler output = subprocess.check_output(args, env=env) self.assertEqual(output.rstrip(), b"False") -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 02:54:15 2015 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 23 Jan 2015 01:54:15 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Only_pass_-E_to_the_child_interpreter_if_our_interpreter?= =?utf-8?q?_was_running_in_that?= Message-ID: <20150123015351.118202.90702@psf.io> https://hg.python.org/cpython/rev/94cade7f6e21 changeset: 94248:94cade7f6e21 parent: 94246:5dc7df4e4143 parent: 94247:7f3ac2ce24ed user: Gregory P. Smith date: Thu Jan 22 17:53:24 2015 -0800 summary: Only pass -E to the child interpreter if our interpreter was running in that mode. Explicitly remove the PYTHONFAULTHANDLER environment variable before launching a child interpreter when its presence would impact the test (the reason -E was being used in the first place). This enables running the test in an environment where other Python environment variables must be set in order for things to run (such as using PYTHONHOME to tell an embedded interpreter where it should think it lives). files: Lib/test/test_faulthandler.py | 22 +++++++++++++++------- 1 files changed, 15 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -260,17 +260,25 @@ def test_disabled_by_default(self): # By default, the module should be disabled code = "import faulthandler; print(faulthandler.is_enabled())" - args = (sys.executable, '-E', '-c', code) - # don't use assert_python_ok() because it always enable faulthandler - output = subprocess.check_output(args) + args = filter(None, (sys.executable, + "-E" if sys.flags.ignore_environment else "", + "-c", code)) + env = os.environ.copy() + env.pop("PYTHONFAULTHANDLER", None) + # don't use assert_python_ok() because it always enables faulthandler + output = subprocess.check_output(args, env=env) self.assertEqual(output.rstrip(), b"False") def test_sys_xoptions(self): # Test python -X faulthandler code = "import faulthandler; print(faulthandler.is_enabled())" - args = (sys.executable, "-E", "-X", "faulthandler", "-c", code) - # don't use assert_python_ok() because it always enable faulthandler - output = subprocess.check_output(args) + args = filter(None, (sys.executable, + "-E" if sys.flags.ignore_environment else "", + "-X", "faulthandler", "-c", code)) + env = os.environ.copy() + env.pop("PYTHONFAULTHANDLER", None) + # don't use assert_python_ok() because it always enables faulthandler + output = subprocess.check_output(args, env=env) self.assertEqual(output.rstrip(), b"True") def test_env_var(self): @@ -279,7 +287,7 @@ args = (sys.executable, "-c", code) env = os.environ.copy() env['PYTHONFAULTHANDLER'] = '' - # don't use assert_python_ok() because it always enable faulthandler + # don't use assert_python_ok() because it always enables faulthandler output = subprocess.check_output(args, env=env) self.assertEqual(output.rstrip(), b"False") -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 07:05:31 2015 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 23 Jan 2015 06:05:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Remove_the_unimplemented_but_ignored_without=3D=27-E=27_?= =?utf-8?q?parameters_being_passed_to?= Message-ID: <20150123060512.118208.64007@psf.io> https://hg.python.org/cpython/rev/110039d91faf changeset: 94250:110039d91faf parent: 94248:94cade7f6e21 parent: 94249:7b833bd1f509 user: Gregory P. Smith date: Thu Jan 22 22:05:00 2015 -0800 summary: Remove the unimplemented but ignored without='-E' parameters being passed to script_helper.assert_python_failure(). No such feature has ever existed, thus it doesn't do what the comment claims. (It does add a 'without' variable to the environment of the child process but that was not intended) files: Lib/test/test_cmd_line.py | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -426,13 +426,11 @@ self.assertIn(b'Unknown option: -z', err) self.assertEqual(err.splitlines().count(b'Unknown option: -z'), 1) self.assertEqual(b'', out) - # Add "without='-E'" to prevent _assert_python to append -E - # to env_vars and change the output of stderr - rc, out, err = assert_python_failure('-z', without='-E') + rc, out, err = assert_python_failure('-z') self.assertIn(b'Unknown option: -z', err) self.assertEqual(err.splitlines().count(b'Unknown option: -z'), 1) self.assertEqual(b'', out) - rc, out, err = assert_python_failure('-a', '-z', without='-E') + rc, out, err = assert_python_failure('-a', '-z') self.assertIn(b'Unknown option: -a', err) # only the first unknown option is reported self.assertNotIn(b'Unknown option: -z', err) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 07:05:33 2015 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 23 Jan 2015 06:05:33 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Remove_the_uni?= =?utf-8?q?mplemented_but_ignored_without=3D=27-E=27_parameters_being_pass?= =?utf-8?q?ed_to?= Message-ID: <20150123060512.87133.76800@psf.io> https://hg.python.org/cpython/rev/7b833bd1f509 changeset: 94249:7b833bd1f509 branch: 3.4 parent: 94247:7f3ac2ce24ed user: Gregory P. Smith date: Thu Jan 22 22:04:16 2015 -0800 summary: Remove the unimplemented but ignored without='-E' parameters being passed to script_helper.assert_python_failure(). No such feature has ever existed, thus it doesn't do what the comment claims. (It does add a 'without' variable to the environment of the child process but that was not intended) files: Lib/test/test_cmd_line.py | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -426,13 +426,11 @@ self.assertIn(b'Unknown option: -z', err) self.assertEqual(err.splitlines().count(b'Unknown option: -z'), 1) self.assertEqual(b'', out) - # Add "without='-E'" to prevent _assert_python to append -E - # to env_vars and change the output of stderr - rc, out, err = assert_python_failure('-z', without='-E') + rc, out, err = assert_python_failure('-z') self.assertIn(b'Unknown option: -z', err) self.assertEqual(err.splitlines().count(b'Unknown option: -z'), 1) self.assertEqual(b'', out) - rc, out, err = assert_python_failure('-a', '-z', without='-E') + rc, out, err = assert_python_failure('-a', '-z') self.assertIn(b'Unknown option: -a', err) # only the first unknown option is reported self.assertNotIn(b'Unknown option: -z', err) -- Repository URL: https://hg.python.org/cpython From greg at krypto.org Fri Jan 23 07:15:00 2015 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 23 Jan 2015 06:15:00 +0000 Subject: [Python-checkins] cpython (3.4): Remove the unimplemented but ignored without='-E' parameters being passed to References: <20150123060512.87133.76800@psf.io> Message-ID: I misinterpreted the side effect this code was triggering. It turns out that any kwarg would have the desired effect as a side effect (as seen in dummyvar='' being passed later in this file for the same purpose). I'm looking at refactoring various uses of -E that interfere with running the test suite in an environment where you need PYTHONHOME set. I will revert this change or otherwise do the equivalent of reinstating its behavior in a followup check-in as appropriate. -gregory.p.smith On Thu Jan 22 2015 at 10:05:56 PM gregory.p.smith < python-checkins at python.org> wrote: > https://hg.python.org/cpython/rev/7b833bd1f509 > changeset: 94249:7b833bd1f509 > branch: 3.4 > parent: 94247:7f3ac2ce24ed > user: Gregory P. Smith > date: Thu Jan 22 22:04:16 2015 -0800 > summary: > Remove the unimplemented but ignored without='-E' parameters being > passed to > script_helper.assert_python_failure(). No such feature has ever existed, > thus it doesn't do what the comment claims. (It does add a 'without' > variable to the environment of the child process but that was not intended) > > files: > Lib/test/test_cmd_line.py | 6 ++---- > 1 files changed, 2 insertions(+), 4 deletions(-) > > > diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py > --- a/Lib/test/test_cmd_line.py > +++ b/Lib/test/test_cmd_line.py > @@ -426,13 +426,11 @@ > self.assertIn(b'Unknown option: -z', err) > self.assertEqual(err.splitlines().count(b'Unknown option: -z'), > 1) > self.assertEqual(b'', out) > - # Add "without='-E'" to prevent _assert_python to append -E > - # to env_vars and change the output of stderr > - rc, out, err = assert_python_failure('-z', without='-E') > + rc, out, err = assert_python_failure('-z') > self.assertIn(b'Unknown option: -z', err) > self.assertEqual(err.splitlines().count(b'Unknown option: -z'), > 1) > self.assertEqual(b'', out) > - rc, out, err = assert_python_failure('-a', '-z', without='-E') > + rc, out, err = assert_python_failure('-a', '-z') > self.assertIn(b'Unknown option: -a', err) > # only the first unknown option is reported > self.assertNotIn(b'Unknown option: -z', err) > > -- > Repository URL: https://hg.python.org/cpython > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-checkins at python.org Fri Jan 23 07:56:22 2015 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 23 Jan 2015 06:56:22 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_revert_7b833bd?= =?utf-8?q?1f509=2E__I_misread_the_side_effect_that_the_code_was_triggerin?= =?utf-8?q?g=2E?= Message-ID: <20150123065619.87109.69874@psf.io> https://hg.python.org/cpython/rev/31982d70a52a changeset: 94251:31982d70a52a branch: 3.4 parent: 94249:7b833bd1f509 user: Gregory P. Smith date: Thu Jan 22 22:55:00 2015 -0800 summary: revert 7b833bd1f509. I misread the side effect that the code was triggering. *any* kwarg supplied to _assert_python causes it to not append -E to the command line flags so without='-E' does effectively work. files: Lib/test/test_cmd_line.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -426,11 +426,13 @@ self.assertIn(b'Unknown option: -z', err) self.assertEqual(err.splitlines().count(b'Unknown option: -z'), 1) self.assertEqual(b'', out) - rc, out, err = assert_python_failure('-z') + # Add "without='-E'" to prevent _assert_python to append -E + # to env_vars and change the output of stderr + rc, out, err = assert_python_failure('-z', without='-E') self.assertIn(b'Unknown option: -z', err) self.assertEqual(err.splitlines().count(b'Unknown option: -z'), 1) self.assertEqual(b'', out) - rc, out, err = assert_python_failure('-a', '-z') + rc, out, err = assert_python_failure('-a', '-z', without='-E') self.assertIn(b'Unknown option: -a', err) # only the first unknown option is reported self.assertNotIn(b'Unknown option: -z', err) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 07:56:22 2015 From: python-checkins at python.org (gregory.p.smith) Date: Fri, 23 Jan 2015 06:56:22 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_revert_7b833bd1f509=2E__I_misread_the_side_effect_that_t?= =?utf-8?q?he_code_was_triggering=2E?= Message-ID: <20150123065619.118226.44262@psf.io> https://hg.python.org/cpython/rev/981ba93bcbde changeset: 94252:981ba93bcbde parent: 94250:110039d91faf parent: 94251:31982d70a52a user: Gregory P. Smith date: Thu Jan 22 22:56:06 2015 -0800 summary: revert 7b833bd1f509. I misread the side effect that the code was triggering. *any* kwarg supplied to _assert_python causes it to not append -E to the command line flags so without='-E' does effectively work. files: Lib/test/test_cmd_line.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -426,11 +426,13 @@ self.assertIn(b'Unknown option: -z', err) self.assertEqual(err.splitlines().count(b'Unknown option: -z'), 1) self.assertEqual(b'', out) - rc, out, err = assert_python_failure('-z') + # Add "without='-E'" to prevent _assert_python to append -E + # to env_vars and change the output of stderr + rc, out, err = assert_python_failure('-z', without='-E') self.assertIn(b'Unknown option: -z', err) self.assertEqual(err.splitlines().count(b'Unknown option: -z'), 1) self.assertEqual(b'', out) - rc, out, err = assert_python_failure('-a', '-z') + rc, out, err = assert_python_failure('-a', '-z', without='-E') self.assertIn(b'Unknown option: -a', err) # only the first unknown option is reported self.assertNotIn(b'Unknown option: -z', err) -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Fri Jan 23 08:47:25 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 23 Jan 2015 08:47:25 +0100 Subject: [Python-checkins] Daily reference leaks (94cade7f6e21): sum=3 Message-ID: results for 94cade7f6e21 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogk25NVu', '-x'] From python-checkins at python.org Fri Jan 23 17:03:18 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 23 Jan 2015 16:03:18 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_http=2Eclient=3A_disable_N?= =?utf-8?q?agle=27s_algorithm_=28closes_=2323302=29?= Message-ID: <20150123160308.84271.20078@psf.io> https://hg.python.org/cpython/rev/42971b769c0b changeset: 94253:42971b769c0b user: Benjamin Peterson date: Fri Jan 23 11:02:57 2015 -0500 summary: http.client: disable Nagle's algorithm (closes #23302) Patch by Demian Brecht. files: Lib/http/client.py | 25 ++++--------------------- Lib/test/test_httplib.py | 25 +++---------------------- Misc/NEWS | 4 ++++ 3 files changed, 11 insertions(+), 43 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -681,14 +681,6 @@ default_port = HTTP_PORT auto_open = 1 debuglevel = 0 - # TCP Maximum Segment Size (MSS) is determined by the TCP stack on - # a per-connection basis. There is no simple and efficient - # platform independent mechanism for determining the MSS, so - # instead a reasonable estimate is chosen. The getsockopt() - # interface using the TCP_MAXSEG parameter may be a suitable - # approach on some operating systems. A value of 16KiB is chosen - # as a reasonable estimate of the maximum MSS. - mss = 16384 def __init__(self, host, port=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, source_address=None): @@ -786,8 +778,9 @@ def connect(self): """Connect to the host and port specified in __init__.""" - self.sock = self._create_connection((self.host,self.port), - self.timeout, self.source_address) + self.sock = self._create_connection( + (self.host,self.port), self.timeout, self.source_address) + self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) if self._tunnel_host: self._tunnel() @@ -866,19 +859,9 @@ self._buffer.extend((b"", b"")) msg = b"\r\n".join(self._buffer) del self._buffer[:] - # If msg and message_body are sent in a single send() call, - # it will avoid performance problems caused by the interaction - # between delayed ack and the Nagle algorithm. However, - # there is no performance gain if the message is larger - # than MSS (and there is a memory penalty for the message - # copy). - if isinstance(message_body, bytes) and len(message_body) < self.mss: - msg += message_body - message_body = None + self.send(msg) if message_body is not None: - # message_body was not a string (i.e. it is a file), and - # we must run the risk of Nagle. self.send(message_body) def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0): diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -70,6 +70,9 @@ def close(self): pass + def setsockopt(self, level, optname, value): + pass + class EPipeSocket(FakeSocket): def __init__(self, text, pipe_trigger): @@ -658,28 +661,6 @@ resp.close() self.assertTrue(resp.closed) - def test_delayed_ack_opt(self): - # Test that Nagle/delayed_ack optimistaion works correctly. - - # For small payloads, it should coalesce the body with - # headers, resulting in a single sendall() call - conn = client.HTTPConnection('example.com') - sock = FakeSocket(None) - conn.sock = sock - body = b'x' * (conn.mss - 1) - conn.request('POST', '/', body) - self.assertEqual(sock.sendall_calls, 1) - - # For large payloads, it should send the headers and - # then the body, resulting in more than one sendall() - # call - conn = client.HTTPConnection('example.com') - sock = FakeSocket(None) - conn.sock = sock - body = b'x' * conn.mss - conn.request('POST', '/', body) - self.assertGreater(sock.sendall_calls, 1) - def test_error_leak(self): # Test that the socket is not leaked if getresponse() fails conn = client.HTTPConnection('example.com') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1308,6 +1308,10 @@ - Issue #22733: Fix ffi_prep_args not zero-extending argument values correctly on 64-bit Windows. +- Issue #23302: Default to TCP_NODELAY=1 upon establishing an HTTPConnection. + Removed use of hard-coded MSS as it's an optimization that's no longer needed + with Nagle disabled. + IDLE ---- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 20:37:19 2015 From: python-checkins at python.org (vinay.sajip) Date: Fri, 23 Jan 2015 19:37:19 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Updated_pyvenv?= =?utf-8?q?_documentation_to_match_its_current_behaviour=2E?= Message-ID: <20150123193710.75774.2717@psf.io> https://hg.python.org/cpython/rev/56f2a42e0897 changeset: 94254:56f2a42e0897 branch: 3.4 parent: 94251:31982d70a52a user: Vinay Sajip date: Fri Jan 23 19:35:12 2015 +0000 summary: Updated pyvenv documentation to match its current behaviour. files: Doc/using/venv-create.inc | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -57,8 +57,11 @@ Installs pip by default, added the ``--without-pip`` and ``--copies`` options -If the target directory already exists an error will be raised, unless -the ``--clear`` or ``--upgrade`` option was provided. +.. versionchanged:: 3.4 + In earlier versions, if the target directory already existed, an error was + raised, unless the ``--clear`` or ``--upgrade`` option was provided. Now, + if an existing directory is specified, its contents are removed and + the directory is processed as if it had been newly created. The created ``pyvenv.cfg`` file also includes the ``include-system-site-packages`` key, set to ``true`` if ``venv`` is -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 20:37:19 2015 From: python-checkins at python.org (vinay.sajip) Date: Fri, 23 Jan 2015 19:37:19 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2323202=3A_pyvenv_documentation_updated_to_match?= =?utf-8?q?_its_behavior=2E?= Message-ID: <20150123193710.75774.57324@psf.io> https://hg.python.org/cpython/rev/a3a44d871d70 changeset: 94255:a3a44d871d70 parent: 94253:42971b769c0b parent: 94254:56f2a42e0897 user: Vinay Sajip date: Fri Jan 23 19:36:54 2015 +0000 summary: Closes #23202: pyvenv documentation updated to match its behavior. files: Doc/using/venv-create.inc | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -57,8 +57,11 @@ Installs pip by default, added the ``--without-pip`` and ``--copies`` options -If the target directory already exists an error will be raised, unless -the ``--clear`` or ``--upgrade`` option was provided. +.. versionchanged:: 3.4 + In earlier versions, if the target directory already existed, an error was + raised, unless the ``--clear`` or ``--upgrade`` option was provided. Now, + if an existing directory is specified, its contents are removed and + the directory is processed as if it had been newly created. The created ``pyvenv.cfg`` file also includes the ``include-system-site-packages`` key, set to ``true`` if ``venv`` is -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 20:54:42 2015 From: python-checkins at python.org (vinay.sajip) Date: Fri, 23 Jan 2015 19:54:42 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjA3?= =?utf-8?q?=3A_logging=2EbasicConfig=28=29_now_does_additional_validation_?= =?utf-8?q?of_its?= Message-ID: <20150123195436.55122.3263@psf.io> https://hg.python.org/cpython/rev/2bc3e839a3a3 changeset: 94256:2bc3e839a3a3 branch: 3.4 parent: 94254:56f2a42e0897 user: Vinay Sajip date: Fri Jan 23 19:52:21 2015 +0000 summary: Issue #23207: logging.basicConfig() now does additional validation of its arguments. files: Lib/logging/__init__.py | 19 +++++++++++-------- 1 files changed, 11 insertions(+), 8 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1723,7 +1723,7 @@ _acquireLock() try: if len(root.handlers) == 0: - handlers = kwargs.get("handlers") + handlers = kwargs.pop("handlers", None) if handlers is None: if "stream" in kwargs and "filename" in kwargs: raise ValueError("'stream' and 'filename' should not be " @@ -1733,28 +1733,31 @@ raise ValueError("'stream' or 'filename' should not be " "specified together with 'handlers'") if handlers is None: - filename = kwargs.get("filename") + filename = kwargs.pop("filename", None) if filename: - mode = kwargs.get("filemode", 'a') + mode = kwargs.pop("filemode", 'a') h = FileHandler(filename, mode) else: - stream = kwargs.get("stream") + stream = kwargs.pop("stream", None) h = StreamHandler(stream) handlers = [h] - dfs = kwargs.get("datefmt", None) - style = kwargs.get("style", '%') + dfs = kwargs.pop("datefmt", None) + style = kwargs.pop("style", '%') if style not in _STYLES: raise ValueError('Style must be one of: %s' % ','.join( _STYLES.keys())) - fs = kwargs.get("format", _STYLES[style][1]) + fs = kwargs.pop("format", _STYLES[style][1]) fmt = Formatter(fs, dfs, style) for h in handlers: if h.formatter is None: h.setFormatter(fmt) root.addHandler(h) - level = kwargs.get("level") + level = kwargs.pop("level", None) if level is not None: root.setLevel(level) + if kwargs: + keys = ', '.join(kwargs.keys()) + raise ValueError('Unrecognised argument(s): %s' % keys) finally: _releaseLock() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 20:54:42 2015 From: python-checkins at python.org (vinay.sajip) Date: Fri, 23 Jan 2015 19:54:42 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2323207=3A_logging=2EbasicConfig=28=29_now_does_?= =?utf-8?q?additional_validation_of_its?= Message-ID: <20150123195436.55098.70095@psf.io> https://hg.python.org/cpython/rev/06ba5e776a6e changeset: 94257:06ba5e776a6e parent: 94255:a3a44d871d70 parent: 94256:2bc3e839a3a3 user: Vinay Sajip date: Fri Jan 23 19:54:23 2015 +0000 summary: Closes #23207: logging.basicConfig() now does additional validation of its arguments. files: Lib/logging/__init__.py | 19 +++++++++++-------- 1 files changed, 11 insertions(+), 8 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1723,7 +1723,7 @@ _acquireLock() try: if len(root.handlers) == 0: - handlers = kwargs.get("handlers") + handlers = kwargs.pop("handlers", None) if handlers is None: if "stream" in kwargs and "filename" in kwargs: raise ValueError("'stream' and 'filename' should not be " @@ -1733,28 +1733,31 @@ raise ValueError("'stream' or 'filename' should not be " "specified together with 'handlers'") if handlers is None: - filename = kwargs.get("filename") + filename = kwargs.pop("filename", None) if filename: - mode = kwargs.get("filemode", 'a') + mode = kwargs.pop("filemode", 'a') h = FileHandler(filename, mode) else: - stream = kwargs.get("stream") + stream = kwargs.pop("stream", None) h = StreamHandler(stream) handlers = [h] - dfs = kwargs.get("datefmt", None) - style = kwargs.get("style", '%') + dfs = kwargs.pop("datefmt", None) + style = kwargs.pop("style", '%') if style not in _STYLES: raise ValueError('Style must be one of: %s' % ','.join( _STYLES.keys())) - fs = kwargs.get("format", _STYLES[style][1]) + fs = kwargs.pop("format", _STYLES[style][1]) fmt = Formatter(fs, dfs, style) for h in handlers: if h.formatter is None: h.setFormatter(fmt) root.addHandler(h) - level = kwargs.get("level") + level = kwargs.pop("level", None) if level is not None: root.setLevel(level) + if kwargs: + keys = ', '.join(kwargs.keys()) + raise ValueError('Unrecognised argument(s): %s' % keys) finally: _releaseLock() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 22:20:15 2015 From: python-checkins at python.org (vinay.sajip) Date: Fri, 23 Jan 2015 21:20:15 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2323305=3A_Merged_documentation_fix_from_3=2E4?= =?utf-8?q?=2E?= Message-ID: <20150123212004.87107.763@psf.io> https://hg.python.org/cpython/rev/67ebf7ae686d changeset: 94260:67ebf7ae686d parent: 94257:06ba5e776a6e parent: 94259:93888975606b user: Vinay Sajip date: Fri Jan 23 21:19:53 2015 +0000 summary: Closes #23305: Merged documentation fix from 3.4. files: Doc/library/logging.handlers.rst | 19 ++++++++++--------- 1 files changed, 10 insertions(+), 9 deletions(-) diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -269,15 +269,16 @@ You can use the *maxBytes* and *backupCount* values to allow the file to :dfn:`rollover` at a predetermined size. When the size is about to be exceeded, the file is closed and a new file is silently opened for output. Rollover occurs - whenever the current log file is nearly *maxBytes* in length; if *maxBytes* is - zero, rollover never occurs. If *backupCount* is non-zero, the system will save - old log files by appending the extensions '.1', '.2' etc., to the filename. For - example, with a *backupCount* of 5 and a base file name of :file:`app.log`, you - would get :file:`app.log`, :file:`app.log.1`, :file:`app.log.2`, up to - :file:`app.log.5`. The file being written to is always :file:`app.log`. When - this file is filled, it is closed and renamed to :file:`app.log.1`, and if files - :file:`app.log.1`, :file:`app.log.2`, etc. exist, then they are renamed to - :file:`app.log.2`, :file:`app.log.3` etc. respectively. + whenever the current log file is nearly *maxBytes* in length; if either of + *maxBytes* or *backupCount* is zero, rollover never occurs. If *backupCount* + is non-zero, the system will save old log files by appending the extensions + '.1', '.2' etc., to the filename. For example, with a *backupCount* of 5 and + a base file name of :file:`app.log`, you would get :file:`app.log`, + :file:`app.log.1`, :file:`app.log.2`, up to :file:`app.log.5`. The file being + written to is always :file:`app.log`. When this file is filled, it is closed + and renamed to :file:`app.log.1`, and if files :file:`app.log.1`, + :file:`app.log.2`, etc. exist, then they are renamed to :file:`app.log.2`, + :file:`app.log.3` etc. respectively. .. method:: doRollover() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 22:20:15 2015 From: python-checkins at python.org (vinay.sajip) Date: Fri, 23 Jan 2015 21:20:15 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMzA1?= =?utf-8?q?=3A_clarified_RotatingFileHandler_documentation=2E?= Message-ID: <20150123212003.75788.9219@psf.io> https://hg.python.org/cpython/rev/0375eb71d75e changeset: 94258:0375eb71d75e branch: 2.7 parent: 94228:2cd0aa44d53c user: Vinay Sajip date: Fri Jan 23 21:18:02 2015 +0000 summary: Issue #23305: clarified RotatingFileHandler documentation. files: Doc/library/logging.handlers.rst | 19 ++++++++++--------- 1 files changed, 10 insertions(+), 9 deletions(-) diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -185,15 +185,16 @@ You can use the *maxBytes* and *backupCount* values to allow the file to :dfn:`rollover` at a predetermined size. When the size is about to be exceeded, the file is closed and a new file is silently opened for output. Rollover occurs - whenever the current log file is nearly *maxBytes* in length; if *maxBytes* is - zero, rollover never occurs. If *backupCount* is non-zero, the system will save - old log files by appending the extensions '.1', '.2' etc., to the filename. For - example, with a *backupCount* of 5 and a base file name of :file:`app.log`, you - would get :file:`app.log`, :file:`app.log.1`, :file:`app.log.2`, up to - :file:`app.log.5`. The file being written to is always :file:`app.log`. When - this file is filled, it is closed and renamed to :file:`app.log.1`, and if files - :file:`app.log.1`, :file:`app.log.2`, etc. exist, then they are renamed to - :file:`app.log.2`, :file:`app.log.3` etc. respectively. + whenever the current log file is nearly *maxBytes* in length; if either of + *maxBytes* or *backupCount* is zero, rollover never occurs. If *backupCount* + is non-zero, the system will save old log files by appending the extensions + '.1', '.2' etc., to the filename. For example, with a *backupCount* of 5 and + a base file name of :file:`app.log`, you would get :file:`app.log`, + :file:`app.log.1`, :file:`app.log.2`, up to :file:`app.log.5`. The file being + written to is always :file:`app.log`. When this file is filled, it is closed + and renamed to :file:`app.log.1`, and if files :file:`app.log.1`, + :file:`app.log.2`, etc. exist, then they are renamed to :file:`app.log.2`, + :file:`app.log.3` etc. respectively. .. versionchanged:: 2.6 *delay* was added. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 22:20:16 2015 From: python-checkins at python.org (vinay.sajip) Date: Fri, 23 Jan 2015 21:20:16 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzA1?= =?utf-8?q?=3A_clarified_RotatingFileHandler_documentation=2E?= Message-ID: <20150123212003.118210.18065@psf.io> https://hg.python.org/cpython/rev/93888975606b changeset: 94259:93888975606b branch: 3.4 parent: 94256:2bc3e839a3a3 user: Vinay Sajip date: Fri Jan 23 21:19:04 2015 +0000 summary: Issue #23305: clarified RotatingFileHandler documentation. files: Doc/library/logging.handlers.rst | 19 ++++++++++--------- 1 files changed, 10 insertions(+), 9 deletions(-) diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -269,15 +269,16 @@ You can use the *maxBytes* and *backupCount* values to allow the file to :dfn:`rollover` at a predetermined size. When the size is about to be exceeded, the file is closed and a new file is silently opened for output. Rollover occurs - whenever the current log file is nearly *maxBytes* in length; if *maxBytes* is - zero, rollover never occurs. If *backupCount* is non-zero, the system will save - old log files by appending the extensions '.1', '.2' etc., to the filename. For - example, with a *backupCount* of 5 and a base file name of :file:`app.log`, you - would get :file:`app.log`, :file:`app.log.1`, :file:`app.log.2`, up to - :file:`app.log.5`. The file being written to is always :file:`app.log`. When - this file is filled, it is closed and renamed to :file:`app.log.1`, and if files - :file:`app.log.1`, :file:`app.log.2`, etc. exist, then they are renamed to - :file:`app.log.2`, :file:`app.log.3` etc. respectively. + whenever the current log file is nearly *maxBytes* in length; if either of + *maxBytes* or *backupCount* is zero, rollover never occurs. If *backupCount* + is non-zero, the system will save old log files by appending the extensions + '.1', '.2' etc., to the filename. For example, with a *backupCount* of 5 and + a base file name of :file:`app.log`, you would get :file:`app.log`, + :file:`app.log.1`, :file:`app.log.2`, up to :file:`app.log.5`. The file being + written to is always :file:`app.log`. When this file is filled, it is closed + and renamed to :file:`app.log.1`, and if files :file:`app.log.1`, + :file:`app.log.2`, etc. exist, then they are renamed to :file:`app.log.2`, + :file:`app.log.3` etc. respectively. .. method:: doRollover() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 22:42:59 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 23 Jan 2015 21:42:59 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_add_support_for_ALPN_=28cl?= =?utf-8?q?oses_=2320188=29?= Message-ID: <20150123214256.84259.82336@psf.io> https://hg.python.org/cpython/rev/be9fe0c66075 changeset: 94261:be9fe0c66075 user: Benjamin Peterson date: Fri Jan 23 16:35:37 2015 -0500 summary: add support for ALPN (closes #20188) files: Doc/library/ssl.rst | 34 +++++++- Lib/ssl.py | 27 ++++++- Lib/test/test_ssl.py | 64 ++++++++++++++- Misc/NEWS | 3 + Modules/_ssl.c | 132 +++++++++++++++++++++++++----- 5 files changed, 231 insertions(+), 29 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -673,6 +673,13 @@ .. versionadded:: 3.3 +.. data:: HAS_ALPN + + Whether the OpenSSL library has built-in support for the *Application-Layer + Protocol Negotiation* TLS extension as described in :rfc:`7301`. + + .. versionadded:: 3.5 + .. data:: HAS_ECDH Whether the OpenSSL library has built-in support for Elliptic Curve-based @@ -959,9 +966,18 @@ .. versionadded:: 3.3 +.. method:: SSLSocket.selected_alpn_protocol() + + Return the protocol that was selected during the TLS handshake. If + :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does + not support ALPN, or if the handshake has not happened yet, ``None`` is + returned. + + .. versionadded:: 3.5 + .. method:: SSLSocket.selected_npn_protocol() - Returns the higher-level protocol that was selected during the TLS/SSL + Return the higher-level protocol that was selected during the TLS/SSL handshake. If :meth:`SSLContext.set_npn_protocols` was not called, or if the other party does not support NPN, or if the handshake has not yet happened, this will return ``None``. @@ -1160,6 +1176,20 @@ when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_alpn_protocols(protocols) + + Specify which protocols the socket should advertise during the SSL/TLS + handshake. It should be a list of ASCII strings, like ``['http/1.1', + 'spdy/2']``, ordered by preference. The selection of a protocol will happen + during the handshake, and will play out according to :rfc:`7301`. After a + successful handshake, the :meth:`SSLSocket.selected_alpn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is + False. + + .. versionadded:: 3.5 + .. method:: SSLContext.set_npn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS @@ -1200,7 +1230,7 @@ Due to the early negotiation phase of the TLS connection, only limited methods and attributes are usable like - :meth:`SSLSocket.selected_npn_protocol` and :attr:`SSLSocket.context`. + :meth:`SSLSocket.selected_alpn_protocol` and :attr:`SSLSocket.context`. :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.cipher` and :meth:`SSLSocket.compress` methods require that the TLS connection has progressed beyond the TLS Client Hello and therefore diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -122,7 +122,7 @@ _import_symbols('ALERT_DESCRIPTION_') _import_symbols('SSL_ERROR_') -from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN +from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN, HAS_ALPN from _ssl import _OPENSSL_API_VERSION @@ -374,6 +374,17 @@ self._set_npn_protocols(protos) + def set_alpn_protocols(self, alpn_protocols): + protos = bytearray() + for protocol in alpn_protocols: + b = bytes(protocol, 'ascii') + if len(b) == 0 or len(b) > 255: + raise SSLError('ALPN protocols must be 1 to 255 in length') + protos.append(len(b)) + protos.extend(b) + + self._set_alpn_protocols(protos) + def _load_windows_store_certs(self, storename, purpose): certs = bytearray() for cert, encoding, trust in enum_certificates(storename): @@ -567,6 +578,13 @@ if _ssl.HAS_NPN: return self._sslobj.selected_npn_protocol() + def selected_alpn_protocol(self): + """Return the currently selected ALPN protocol as a string, or ``None`` + if a next protocol was not negotiated or if ALPN is not supported by one + of the peers.""" + if _ssl.HAS_ALPN: + return self._sslobj.selected_alpn_protocol() + def cipher(self): """Return the currently selected cipher as a 3-tuple ``(name, ssl_version, secret_bits)``.""" @@ -783,6 +801,13 @@ else: return self._sslobj.selected_npn_protocol() + def selected_alpn_protocol(self): + self._checkClosed() + if not self._sslobj or not _ssl.HAS_ALPN: + return None + else: + return self._sslobj.selected_alpn_protocol() + def cipher(self): self._checkClosed() if not self._sslobj: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1761,7 +1761,8 @@ try: self.sslconn = self.server.context.wrap_socket( self.sock, server_side=True) - self.server.selected_protocols.append(self.sslconn.selected_npn_protocol()) + self.server.selected_npn_protocols.append(self.sslconn.selected_npn_protocol()) + self.server.selected_alpn_protocols.append(self.sslconn.selected_alpn_protocol()) except (ssl.SSLError, ConnectionResetError) as e: # We treat ConnectionResetError as though it were an # SSLError - OpenSSL on Ubuntu abruptly closes the @@ -1869,7 +1870,8 @@ def __init__(self, certificate=None, ssl_version=None, certreqs=None, cacerts=None, chatty=True, connectionchatty=False, starttls_server=False, - npn_protocols=None, ciphers=None, context=None): + npn_protocols=None, alpn_protocols=None, + ciphers=None, context=None): if context: self.context = context else: @@ -1884,6 +1886,8 @@ self.context.load_cert_chain(certificate) if npn_protocols: self.context.set_npn_protocols(npn_protocols) + if alpn_protocols: + self.context.set_alpn_protocols(alpn_protocols) if ciphers: self.context.set_ciphers(ciphers) self.chatty = chatty @@ -1893,7 +1897,8 @@ self.port = support.bind_port(self.sock) self.flag = None self.active = False - self.selected_protocols = [] + self.selected_npn_protocols = [] + self.selected_alpn_protocols = [] self.shared_ciphers = [] self.conn_errors = [] threading.Thread.__init__(self) @@ -2120,11 +2125,13 @@ 'compression': s.compression(), 'cipher': s.cipher(), 'peercert': s.getpeercert(), + 'client_alpn_protocol': s.selected_alpn_protocol(), 'client_npn_protocol': s.selected_npn_protocol(), 'version': s.version(), }) s.close() - stats['server_npn_protocols'] = server.selected_protocols + stats['server_alpn_protocols'] = server.selected_alpn_protocols + stats['server_npn_protocols'] = server.selected_npn_protocols stats['server_shared_ciphers'] = server.shared_ciphers return stats @@ -3022,6 +3029,55 @@ if "ADH" not in parts and "EDH" not in parts and "DHE" not in parts: self.fail("Non-DH cipher: " + cipher[0]) + def test_selected_alpn_protocol(self): + # selected_alpn_protocol() is None unless ALPN is used. + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.load_cert_chain(CERTFILE) + stats = server_params_test(context, context, + chatty=True, connectionchatty=True) + self.assertIs(stats['client_alpn_protocol'], None) + + @unittest.skipUnless(ssl.HAS_ALPN, "ALPN support required") + def test_selected_alpn_protocol_if_server_uses_alpn(self): + # selected_alpn_protocol() is None unless ALPN is used by the client. + client_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + client_context.load_verify_locations(CERTFILE) + server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + server_context.load_cert_chain(CERTFILE) + server_context.set_alpn_protocols(['foo', 'bar']) + stats = server_params_test(client_context, server_context, + chatty=True, connectionchatty=True) + self.assertIs(stats['client_alpn_protocol'], None) + + @unittest.skipUnless(ssl.HAS_ALPN, "ALPN support needed for this test") + def test_alpn_protocols(self): + server_protocols = ['foo', 'bar', 'milkshake'] + protocol_tests = [ + (['foo', 'bar'], 'foo'), + (['bar', 'foo'], 'bar'), + (['milkshake'], 'milkshake'), + (['http/3.0', 'http/4.0'], 'foo') + ] + for client_protocols, expected in protocol_tests: + server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + server_context.load_cert_chain(CERTFILE) + server_context.set_alpn_protocols(server_protocols) + client_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + client_context.load_cert_chain(CERTFILE) + client_context.set_alpn_protocols(client_protocols) + stats = server_params_test(client_context, server_context, + chatty=True, connectionchatty=True) + + msg = "failed trying %s (s) and %s (c).\n" \ + "was expecting %s, but got %%s from the %%s" \ + % (str(server_protocols), str(client_protocols), + str(expected)) + client_result = stats['client_alpn_protocol'] + self.assertEqual(client_result, expected, msg % (client_result, "client")) + server_result = stats['server_alpn_protocols'][-1] \ + if len(stats['server_alpn_protocols']) else 'nothing' + self.assertEqual(server_result, expected, msg % (server_result, "server")) + def test_selected_npn_protocol(self): # selected_npn_protocol() is None unless NPN is used context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -203,6 +203,9 @@ Library ------- +- Issue #20188: Support Application-Layer Protocol Negotiation (ALPN) in the ssl + module. + - Issue #23133: Pickling of ipaddress objects now produces more compact and portable representation. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -109,6 +109,11 @@ # define HAVE_SNI 0 #endif +/* ALPN added in OpenSSL 1.0.2 */ +#if OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) +# define HAVE_ALPN +#endif + enum py_ssl_error { /* these mirror ssl.h */ PY_SSL_ERROR_NONE, @@ -180,9 +185,13 @@ PyObject_HEAD SSL_CTX *ctx; #ifdef OPENSSL_NPN_NEGOTIATED - char *npn_protocols; + unsigned char *npn_protocols; int npn_protocols_len; #endif +#ifdef HAVE_ALPN + unsigned char *alpn_protocols; + int alpn_protocols_len; +#endif #ifndef OPENSSL_NO_TLSEXT PyObject *set_hostname; #endif @@ -1460,7 +1469,20 @@ if (out == NULL) Py_RETURN_NONE; - return PyUnicode_FromStringAndSize((char *) out, outlen); + return PyUnicode_FromStringAndSize((char *)out, outlen); +} +#endif + +#ifdef HAVE_ALPN +static PyObject *PySSL_selected_alpn_protocol(PySSLSocket *self) { + const unsigned char *out; + unsigned int outlen; + + SSL_get0_alpn_selected(self->ssl, &out, &outlen); + + if (out == NULL) + Py_RETURN_NONE; + return PyUnicode_FromStringAndSize((char *)out, outlen); } #endif @@ -2054,6 +2076,9 @@ #ifdef OPENSSL_NPN_NEGOTIATED {"selected_npn_protocol", (PyCFunction)PySSL_selected_npn_protocol, METH_NOARGS}, #endif +#ifdef HAVE_ALPN + {"selected_alpn_protocol", (PyCFunction)PySSL_selected_alpn_protocol, METH_NOARGS}, +#endif {"compression", (PyCFunction)PySSL_compression, METH_NOARGS}, {"shutdown", (PyCFunction)PySSL_SSLshutdown, METH_NOARGS, PySSL_SSLshutdown_doc}, @@ -2159,6 +2184,9 @@ #ifdef OPENSSL_NPN_NEGOTIATED self->npn_protocols = NULL; #endif +#ifdef HAVE_ALPN + self->alpn_protocols = NULL; +#endif #ifndef OPENSSL_NO_TLSEXT self->set_hostname = NULL; #endif @@ -2218,7 +2246,10 @@ context_clear(self); SSL_CTX_free(self->ctx); #ifdef OPENSSL_NPN_NEGOTIATED - PyMem_Free(self->npn_protocols); + PyMem_FREE(self->npn_protocols); +#endif +#ifdef HAVE_ALPN + PyMem_FREE(self->alpn_protocols); #endif Py_TYPE(self)->tp_free(self); } @@ -2244,6 +2275,23 @@ Py_RETURN_NONE; } +static int +do_protocol_selection(unsigned char **out, unsigned char *outlen, + const unsigned char *remote_protocols, unsigned int remote_protocols_len, + unsigned char *our_protocols, unsigned int our_protocols_len) +{ + if (our_protocols == NULL) { + our_protocols = (unsigned char*)""; + our_protocols_len = 0; + } + + SSL_select_next_proto(out, outlen, + remote_protocols, remote_protocols_len, + our_protocols, our_protocols_len); + + return SSL_TLSEXT_ERR_OK; +} + #ifdef OPENSSL_NPN_NEGOTIATED /* this callback gets passed to SSL_CTX_set_next_protos_advertise_cb */ static int @@ -2254,10 +2302,10 @@ PySSLContext *ssl_ctx = (PySSLContext *) args; if (ssl_ctx->npn_protocols == NULL) { - *data = (unsigned char *) ""; + *data = (unsigned char *)""; *len = 0; } else { - *data = (unsigned char *) ssl_ctx->npn_protocols; + *data = ssl_ctx->npn_protocols; *len = ssl_ctx->npn_protocols_len; } @@ -2270,23 +2318,9 @@ const unsigned char *server, unsigned int server_len, void *args) { - PySSLContext *ssl_ctx = (PySSLContext *) args; - - unsigned char *client = (unsigned char *) ssl_ctx->npn_protocols; - int client_len; - - if (client == NULL) { - client = (unsigned char *) ""; - client_len = 0; - } else { - client_len = ssl_ctx->npn_protocols_len; - } - - SSL_select_next_proto(out, outlen, - server, server_len, - client, client_len); - - return SSL_TLSEXT_ERR_OK; + PySSLContext *ctx = (PySSLContext *)args; + return do_protocol_selection(out, outlen, server, server_len, + ctx->npn_protocols, ctx->npn_protocols_len); } #endif @@ -2329,6 +2363,50 @@ #endif } +#ifdef HAVE_ALPN +static int +_selectALPN_cb(SSL *s, + const unsigned char **out, unsigned char *outlen, + const unsigned char *client_protocols, unsigned int client_protocols_len, + void *args) +{ + PySSLContext *ctx = (PySSLContext *)args; + return do_protocol_selection((unsigned char **)out, outlen, + client_protocols, client_protocols_len, + ctx->alpn_protocols, ctx->alpn_protocols_len); +} +#endif + +static PyObject * +_set_alpn_protocols(PySSLContext *self, PyObject *args) +{ +#ifdef HAVE_ALPN + Py_buffer protos; + + if (!PyArg_ParseTuple(args, "y*:set_npn_protocols", &protos)) + return NULL; + + PyMem_FREE(self->alpn_protocols); + self->alpn_protocols = PyMem_Malloc(protos.len); + if (!self->alpn_protocols) + return PyErr_NoMemory(); + memcpy(self->alpn_protocols, protos.buf, protos.len); + self->alpn_protocols_len = protos.len; + PyBuffer_Release(&protos); + + if (SSL_CTX_set_alpn_protos(self->ctx, self->alpn_protocols, self->alpn_protocols_len)) + return PyErr_NoMemory(); + SSL_CTX_set_alpn_select_cb(self->ctx, _selectALPN_cb, self); + + PyBuffer_Release(&protos); + Py_RETURN_NONE; +#else + PyErr_SetString(PyExc_NotImplementedError, + "The ALPN extension requires OpenSSL 1.0.2 or later."); + return NULL; +#endif +} + static PyObject * get_verify_mode(PySSLContext *self, void *c) { @@ -3307,6 +3385,8 @@ METH_VARARGS | METH_KEYWORDS, NULL}, {"set_ciphers", (PyCFunction) set_ciphers, METH_VARARGS, NULL}, + {"_set_alpn_protocols", (PyCFunction) _set_alpn_protocols, + METH_VARARGS, NULL}, {"_set_npn_protocols", (PyCFunction) _set_npn_protocols, METH_VARARGS, NULL}, {"load_cert_chain", (PyCFunction) load_cert_chain, @@ -4502,6 +4582,14 @@ Py_INCREF(r); PyModule_AddObject(m, "HAS_NPN", r); +#ifdef HAVE_ALPN + r = Py_True; +#else + r = Py_False; +#endif + Py_INCREF(r); + PyModule_AddObject(m, "HAS_ALPN", r); + /* Mappings for error codes */ err_codes_to_names = PyDict_New(); err_names_to_codes = PyDict_New(); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 22:42:59 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 23 Jan 2015 21:42:59 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_pep_466_backpo?= =?utf-8?q?rt_of_alpn_=28=2320188=29?= Message-ID: <20150123214257.118212.81892@psf.io> https://hg.python.org/cpython/rev/7ce67d3f0908 changeset: 94262:7ce67d3f0908 branch: 2.7 parent: 94258:0375eb71d75e user: Benjamin Peterson date: Fri Jan 23 16:35:37 2015 -0500 summary: pep 466 backport of alpn (#20188) files: Doc/library/ssl.rst | 34 +++++++- Lib/ssl.py | 20 ++++- Lib/test/test_ssl.py | 64 ++++++++++++++- Misc/NEWS | 3 + Modules/_ssl.c | 132 +++++++++++++++++++++++++----- 5 files changed, 224 insertions(+), 29 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -638,6 +638,13 @@ .. versionadded:: 2.7.9 +.. data:: HAS_ALPN + + Whether the OpenSSL library has built-in support for the *Application-Layer + Protocol Negotiation* TLS extension as described in :rfc:`7301`. + + .. versionadded:: 3.5 + .. data:: HAS_ECDH Whether the OpenSSL library has built-in support for Elliptic Curve-based @@ -864,9 +871,18 @@ .. versionadded:: 2.7.9 +.. method:: SSLSocket.selected_alpn_protocol() + + Return the protocol that was selected during the TLS handshake. If + :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does + not support ALPN, or if the handshake has not happened yet, ``None`` is + returned. + + .. versionadded:: 3.5 + .. method:: SSLSocket.selected_npn_protocol() - Returns the higher-level protocol that was selected during the TLS/SSL + Return the higher-level protocol that was selected during the TLS/SSL handshake. If :meth:`SSLContext.set_npn_protocols` was not called, or if the other party does not support NPN, or if the handshake has not yet happened, this will return ``None``. @@ -1034,6 +1050,20 @@ when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_alpn_protocols(protocols) + + Specify which protocols the socket should advertise during the SSL/TLS + handshake. It should be a list of ASCII strings, like ``['http/1.1', + 'spdy/2']``, ordered by preference. The selection of a protocol will happen + during the handshake, and will play out according to :rfc:`7301`. After a + successful handshake, the :meth:`SSLSocket.selected_alpn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is + False. + + .. versionadded:: 3.5 + .. method:: SSLContext.set_npn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS @@ -1072,7 +1102,7 @@ Due to the early negotiation phase of the TLS connection, only limited methods and attributes are usable like - :meth:`SSLSocket.selected_npn_protocol` and :attr:`SSLSocket.context`. + :meth:`SSLSocket.selected_alpn_protocol` and :attr:`SSLSocket.context`. :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.cipher` and :meth:`SSLSocket.compress` methods require that the TLS connection has progressed beyond the TLS Client Hello and therefore diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -123,7 +123,7 @@ _import_symbols('SSL_ERROR_') _import_symbols('PROTOCOL_') -from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN +from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN, HAS_ALPN from _ssl import _OPENSSL_API_VERSION @@ -365,6 +365,17 @@ self._set_npn_protocols(protos) + def set_alpn_protocols(self, alpn_protocols): + protos = bytearray() + for protocol in alpn_protocols: + b = protocol.encode('ascii') + if len(b) == 0 or len(b) > 255: + raise SSLError('ALPN protocols must be 1 to 255 in length') + protos.append(len(b)) + protos.extend(b) + + self._set_alpn_protocols(protos) + def _load_windows_store_certs(self, storename, purpose): certs = bytearray() for cert, encoding, trust in enum_certificates(storename): @@ -647,6 +658,13 @@ else: return self._sslobj.selected_npn_protocol() + def selected_alpn_protocol(self): + self._checkClosed() + if not self._sslobj or not _ssl.HAS_ALPN: + return None + else: + return self._sslobj.selected_alpn_protocol() + def cipher(self): self._checkClosed() if not self._sslobj: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1569,7 +1569,8 @@ try: self.sslconn = self.server.context.wrap_socket( self.sock, server_side=True) - self.server.selected_protocols.append(self.sslconn.selected_npn_protocol()) + self.server.selected_npn_protocols.append(self.sslconn.selected_npn_protocol()) + self.server.selected_alpn_protocols.append(self.sslconn.selected_alpn_protocol()) except socket.error as e: # We treat ConnectionResetError as though it were an # SSLError - OpenSSL on Ubuntu abruptly closes the @@ -1678,7 +1679,8 @@ def __init__(self, certificate=None, ssl_version=None, certreqs=None, cacerts=None, chatty=True, connectionchatty=False, starttls_server=False, - npn_protocols=None, ciphers=None, context=None): + npn_protocols=None, alpn_protocols=None, + ciphers=None, context=None): if context: self.context = context else: @@ -1693,6 +1695,8 @@ self.context.load_cert_chain(certificate) if npn_protocols: self.context.set_npn_protocols(npn_protocols) + if alpn_protocols: + self.context.set_alpn_protocols(alpn_protocols) if ciphers: self.context.set_ciphers(ciphers) self.chatty = chatty @@ -1702,7 +1706,8 @@ self.port = support.bind_port(self.sock) self.flag = None self.active = False - self.selected_protocols = [] + self.selected_npn_protocols = [] + self.selected_alpn_protocols = [] self.conn_errors = [] threading.Thread.__init__(self) self.daemon = True @@ -1927,11 +1932,13 @@ 'compression': s.compression(), 'cipher': s.cipher(), 'peercert': s.getpeercert(), + 'client_alpn_protocol': s.selected_alpn_protocol(), 'client_npn_protocol': s.selected_npn_protocol(), 'version': s.version(), }) s.close() - stats['server_npn_protocols'] = server.selected_protocols + stats['server_alpn_protocols'] = server.selected_alpn_protocols + stats['server_npn_protocols'] = server.selected_npn_protocols return stats def try_protocol_combo(server_protocol, client_protocol, expect_success, @@ -2787,6 +2794,55 @@ if "ADH" not in parts and "EDH" not in parts and "DHE" not in parts: self.fail("Non-DH cipher: " + cipher[0]) + def test_selected_alpn_protocol(self): + # selected_alpn_protocol() is None unless ALPN is used. + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.load_cert_chain(CERTFILE) + stats = server_params_test(context, context, + chatty=True, connectionchatty=True) + self.assertIs(stats['client_alpn_protocol'], None) + + @unittest.skipUnless(ssl.HAS_ALPN, "ALPN support required") + def test_selected_alpn_protocol_if_server_uses_alpn(self): + # selected_alpn_protocol() is None unless ALPN is used by the client. + client_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + client_context.load_verify_locations(CERTFILE) + server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + server_context.load_cert_chain(CERTFILE) + server_context.set_alpn_protocols(['foo', 'bar']) + stats = server_params_test(client_context, server_context, + chatty=True, connectionchatty=True) + self.assertIs(stats['client_alpn_protocol'], None) + + @unittest.skipUnless(ssl.HAS_ALPN, "ALPN support needed for this test") + def test_alpn_protocols(self): + server_protocols = ['foo', 'bar', 'milkshake'] + protocol_tests = [ + (['foo', 'bar'], 'foo'), + (['bar', 'foo'], 'bar'), + (['milkshake'], 'milkshake'), + (['http/3.0', 'http/4.0'], 'foo') + ] + for client_protocols, expected in protocol_tests: + server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + server_context.load_cert_chain(CERTFILE) + server_context.set_alpn_protocols(server_protocols) + client_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + client_context.load_cert_chain(CERTFILE) + client_context.set_alpn_protocols(client_protocols) + stats = server_params_test(client_context, server_context, + chatty=True, connectionchatty=True) + + msg = "failed trying %s (s) and %s (c).\n" \ + "was expecting %s, but got %%s from the %%s" \ + % (str(server_protocols), str(client_protocols), + str(expected)) + client_result = stats['client_alpn_protocol'] + self.assertEqual(client_result, expected, msg % (client_result, "client")) + server_result = stats['server_alpn_protocols'][-1] \ + if len(stats['server_alpn_protocols']) else 'nothing' + self.assertEqual(server_result, expected, msg % (server_result, "server")) + def test_selected_npn_protocol(self): # selected_npn_protocol() is None unless NPN is used context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,9 @@ Library ------- +- Issue #20188: Support Application-Layer Protocol Negotiation (ALPN) in the ssl + module. + - Issue #23248: Update ssl error codes from latest OpenSSL git master. - Issue #23098: 64-bit dev_t is now supported in the os module. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -105,6 +105,11 @@ # define HAVE_SNI 0 #endif +/* ALPN added in OpenSSL 1.0.2 */ +#if OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) +# define HAVE_ALPN +#endif + enum py_ssl_error { /* these mirror ssl.h */ PY_SSL_ERROR_NONE, @@ -205,9 +210,13 @@ PyObject_HEAD SSL_CTX *ctx; #ifdef OPENSSL_NPN_NEGOTIATED - char *npn_protocols; + unsigned char *npn_protocols; int npn_protocols_len; #endif +#ifdef HAVE_ALPN + unsigned char *alpn_protocols; + int alpn_protocols_len; +#endif #ifndef OPENSSL_NO_TLSEXT PyObject *set_hostname; #endif @@ -1408,7 +1417,20 @@ if (out == NULL) Py_RETURN_NONE; - return PyUnicode_FromStringAndSize((char *) out, outlen); + return PyString_FromStringAndSize((char *)out, outlen); +} +#endif + +#ifdef HAVE_ALPN +static PyObject *PySSL_selected_alpn_protocol(PySSLSocket *self) { + const unsigned char *out; + unsigned int outlen; + + SSL_get0_alpn_selected(self->ssl, &out, &outlen); + + if (out == NULL) + Py_RETURN_NONE; + return PyString_FromStringAndSize((char *)out, outlen); } #endif @@ -1925,6 +1947,9 @@ #ifdef OPENSSL_NPN_NEGOTIATED {"selected_npn_protocol", (PyCFunction)PySSL_selected_npn_protocol, METH_NOARGS}, #endif +#ifdef HAVE_ALPN + {"selected_alpn_protocol", (PyCFunction)PySSL_selected_alpn_protocol, METH_NOARGS}, +#endif {"compression", (PyCFunction)PySSL_compression, METH_NOARGS}, {"shutdown", (PyCFunction)PySSL_SSLshutdown, METH_NOARGS, PySSL_SSLshutdown_doc}, @@ -2032,6 +2057,9 @@ #ifdef OPENSSL_NPN_NEGOTIATED self->npn_protocols = NULL; #endif +#ifdef HAVE_ALPN + self->alpn_protocols = NULL; +#endif #ifndef OPENSSL_NO_TLSEXT self->set_hostname = NULL; #endif @@ -2091,7 +2119,10 @@ context_clear(self); SSL_CTX_free(self->ctx); #ifdef OPENSSL_NPN_NEGOTIATED - PyMem_Free(self->npn_protocols); + PyMem_FREE(self->npn_protocols); +#endif +#ifdef HAVE_ALPN + PyMem_FREE(self->alpn_protocols); #endif Py_TYPE(self)->tp_free(self); } @@ -2117,6 +2148,23 @@ Py_RETURN_NONE; } +static int +do_protocol_selection(unsigned char **out, unsigned char *outlen, + const unsigned char *remote_protocols, unsigned int remote_protocols_len, + unsigned char *our_protocols, unsigned int our_protocols_len) +{ + if (our_protocols == NULL) { + our_protocols = (unsigned char*)""; + our_protocols_len = 0; + } + + SSL_select_next_proto(out, outlen, + remote_protocols, remote_protocols_len, + our_protocols, our_protocols_len); + + return SSL_TLSEXT_ERR_OK; +} + #ifdef OPENSSL_NPN_NEGOTIATED /* this callback gets passed to SSL_CTX_set_next_protos_advertise_cb */ static int @@ -2127,10 +2175,10 @@ PySSLContext *ssl_ctx = (PySSLContext *) args; if (ssl_ctx->npn_protocols == NULL) { - *data = (unsigned char *) ""; + *data = (unsigned char *)""; *len = 0; } else { - *data = (unsigned char *) ssl_ctx->npn_protocols; + *data = ssl_ctx->npn_protocols; *len = ssl_ctx->npn_protocols_len; } @@ -2143,23 +2191,9 @@ const unsigned char *server, unsigned int server_len, void *args) { - PySSLContext *ssl_ctx = (PySSLContext *) args; - - unsigned char *client = (unsigned char *) ssl_ctx->npn_protocols; - int client_len; - - if (client == NULL) { - client = (unsigned char *) ""; - client_len = 0; - } else { - client_len = ssl_ctx->npn_protocols_len; - } - - SSL_select_next_proto(out, outlen, - server, server_len, - client, client_len); - - return SSL_TLSEXT_ERR_OK; + PySSLContext *ctx = (PySSLContext *)args; + return do_protocol_selection(out, outlen, server, server_len, + ctx->npn_protocols, ctx->npn_protocols_len); } #endif @@ -2202,6 +2236,50 @@ #endif } +#ifdef HAVE_ALPN +static int +_selectALPN_cb(SSL *s, + const unsigned char **out, unsigned char *outlen, + const unsigned char *client_protocols, unsigned int client_protocols_len, + void *args) +{ + PySSLContext *ctx = (PySSLContext *)args; + return do_protocol_selection((unsigned char **)out, outlen, + client_protocols, client_protocols_len, + ctx->alpn_protocols, ctx->alpn_protocols_len); +} +#endif + +static PyObject * +_set_alpn_protocols(PySSLContext *self, PyObject *args) +{ +#ifdef HAVE_ALPN + Py_buffer protos; + + if (!PyArg_ParseTuple(args, "s*:set_npn_protocols", &protos)) + return NULL; + + PyMem_FREE(self->alpn_protocols); + self->alpn_protocols = PyMem_Malloc(protos.len); + if (!self->alpn_protocols) + return PyErr_NoMemory(); + memcpy(self->alpn_protocols, protos.buf, protos.len); + self->alpn_protocols_len = protos.len; + PyBuffer_Release(&protos); + + if (SSL_CTX_set_alpn_protos(self->ctx, self->alpn_protocols, self->alpn_protocols_len)) + return PyErr_NoMemory(); + SSL_CTX_set_alpn_select_cb(self->ctx, _selectALPN_cb, self); + + PyBuffer_Release(&protos); + Py_RETURN_NONE; +#else + PyErr_SetString(PyExc_NotImplementedError, + "The ALPN extension requires OpenSSL 1.0.2 or later."); + return NULL; +#endif +} + static PyObject * get_verify_mode(PySSLContext *self, void *c) { @@ -3188,6 +3266,8 @@ METH_VARARGS | METH_KEYWORDS, NULL}, {"set_ciphers", (PyCFunction) set_ciphers, METH_VARARGS, NULL}, + {"_set_alpn_protocols", (PyCFunction) _set_alpn_protocols, + METH_VARARGS, NULL}, {"_set_npn_protocols", (PyCFunction) _set_npn_protocols, METH_VARARGS, NULL}, {"load_cert_chain", (PyCFunction) load_cert_chain, @@ -4100,6 +4180,14 @@ Py_INCREF(r); PyModule_AddObject(m, "HAS_NPN", r); +#ifdef HAVE_ALPN + r = Py_True; +#else + r = Py_False; +#endif + Py_INCREF(r); + PyModule_AddObject(m, "HAS_ALPN", r); + /* Mappings for error codes */ err_codes_to_names = PyDict_New(); err_names_to_codes = PyDict_New(); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 22:48:07 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 23 Jan 2015 21:48:07 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_fix_versioncha?= =?utf-8?q?nged?= Message-ID: <20150123214759.55126.30500@psf.io> https://hg.python.org/cpython/rev/f3ec1793f170 changeset: 94263:f3ec1793f170 branch: 2.7 user: Benjamin Peterson date: Fri Jan 23 16:47:52 2015 -0500 summary: fix versionchanged files: Doc/library/ssl.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -643,7 +643,7 @@ Whether the OpenSSL library has built-in support for the *Application-Layer Protocol Negotiation* TLS extension as described in :rfc:`7301`. - .. versionadded:: 3.5 + .. versionadded:: 2.7.10 .. data:: HAS_ECDH @@ -878,7 +878,7 @@ not support ALPN, or if the handshake has not happened yet, ``None`` is returned. - .. versionadded:: 3.5 + .. versionadded:: 2.7.10 .. method:: SSLSocket.selected_npn_protocol() @@ -1062,7 +1062,7 @@ This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is False. - .. versionadded:: 3.5 + .. versionadded:: 2.7.10 .. method:: SSLContext.set_npn_protocols(protocols) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 23:30:38 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 23 Jan 2015 22:30:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_prefer_server_alpn_orderin?= =?utf-8?q?g_over_the_client=27s?= Message-ID: <20150123223035.118214.8440@psf.io> https://hg.python.org/cpython/rev/eaa38b75cc78 changeset: 94264:eaa38b75cc78 parent: 94261:be9fe0c66075 user: Benjamin Peterson date: Fri Jan 23 17:30:26 2015 -0500 summary: prefer server alpn ordering over the client's files: Doc/library/ssl.rst | 3 +- Lib/test/test_ssl.py | 4 +- Modules/_ssl.c | 35 +++++++++++++++++++------------ 3 files changed, 25 insertions(+), 17 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -970,7 +970,8 @@ Return the protocol that was selected during the TLS handshake. If :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does - not support ALPN, or if the handshake has not happened yet, ``None`` is + not support ALPN, if this socket does not support any of the client's + proposed protocols, or if the handshake has not happened yet, ``None`` is returned. .. versionadded:: 3.5 diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3054,9 +3054,9 @@ server_protocols = ['foo', 'bar', 'milkshake'] protocol_tests = [ (['foo', 'bar'], 'foo'), - (['bar', 'foo'], 'bar'), + (['bar', 'foo'], 'foo'), (['milkshake'], 'milkshake'), - (['http/3.0', 'http/4.0'], 'foo') + (['http/3.0', 'http/4.0'], None) ] for client_protocols, expected in protocol_tests: server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2276,18 +2276,25 @@ } static int -do_protocol_selection(unsigned char **out, unsigned char *outlen, - const unsigned char *remote_protocols, unsigned int remote_protocols_len, - unsigned char *our_protocols, unsigned int our_protocols_len) +do_protocol_selection(int alpn, unsigned char **out, unsigned char *outlen, + const unsigned char *server_protocols, unsigned int server_protocols_len, + const unsigned char *client_protocols, unsigned int client_protocols_len) { - if (our_protocols == NULL) { - our_protocols = (unsigned char*)""; - our_protocols_len = 0; + int ret; + if (client_protocols == NULL) { + client_protocols = (unsigned char *)""; + client_protocols_len = 0; } - - SSL_select_next_proto(out, outlen, - remote_protocols, remote_protocols_len, - our_protocols, our_protocols_len); + if (server_protocols == NULL) { + server_protocols = (unsigned char *)""; + server_protocols_len = 0; + } + + ret = SSL_select_next_proto(out, outlen, + server_protocols, server_protocols_len, + client_protocols, client_protocols_len); + if (alpn && ret != OPENSSL_NPN_NEGOTIATED) + return SSL_TLSEXT_ERR_NOACK; return SSL_TLSEXT_ERR_OK; } @@ -2319,7 +2326,7 @@ void *args) { PySSLContext *ctx = (PySSLContext *)args; - return do_protocol_selection(out, outlen, server, server_len, + return do_protocol_selection(0, out, outlen, server, server_len, ctx->npn_protocols, ctx->npn_protocols_len); } #endif @@ -2371,9 +2378,9 @@ void *args) { PySSLContext *ctx = (PySSLContext *)args; - return do_protocol_selection((unsigned char **)out, outlen, - client_protocols, client_protocols_len, - ctx->alpn_protocols, ctx->alpn_protocols_len); + return do_protocol_selection(1, (unsigned char **)out, outlen, + ctx->alpn_protocols, ctx->alpn_protocols_len, + client_protocols, client_protocols_len); } #endif -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 23 23:30:38 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 23 Jan 2015 22:30:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_prefer_server_?= =?utf-8?q?alpn_ordering_over_the_client=27s?= Message-ID: <20150123223035.84253.51300@psf.io> https://hg.python.org/cpython/rev/94ec4d8cf104 changeset: 94265:94ec4d8cf104 branch: 2.7 parent: 94263:f3ec1793f170 user: Benjamin Peterson date: Fri Jan 23 17:30:26 2015 -0500 summary: prefer server alpn ordering over the client's files: Doc/library/ssl.rst | 3 +- Lib/test/test_ssl.py | 4 +- Modules/_ssl.c | 35 +++++++++++++++++++------------ 3 files changed, 25 insertions(+), 17 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -875,7 +875,8 @@ Return the protocol that was selected during the TLS handshake. If :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does - not support ALPN, or if the handshake has not happened yet, ``None`` is + not support ALPN, if this socket does not support any of the client's + proposed protocols, or if the handshake has not happened yet, ``None`` is returned. .. versionadded:: 2.7.10 diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2819,9 +2819,9 @@ server_protocols = ['foo', 'bar', 'milkshake'] protocol_tests = [ (['foo', 'bar'], 'foo'), - (['bar', 'foo'], 'bar'), + (['bar', 'foo'], 'foo'), (['milkshake'], 'milkshake'), - (['http/3.0', 'http/4.0'], 'foo') + (['http/3.0', 'http/4.0'], None) ] for client_protocols, expected in protocol_tests: server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2149,18 +2149,25 @@ } static int -do_protocol_selection(unsigned char **out, unsigned char *outlen, - const unsigned char *remote_protocols, unsigned int remote_protocols_len, - unsigned char *our_protocols, unsigned int our_protocols_len) +do_protocol_selection(int alpn, unsigned char **out, unsigned char *outlen, + const unsigned char *server_protocols, unsigned int server_protocols_len, + const unsigned char *client_protocols, unsigned int client_protocols_len) { - if (our_protocols == NULL) { - our_protocols = (unsigned char*)""; - our_protocols_len = 0; + int ret; + if (client_protocols == NULL) { + client_protocols = (unsigned char *)""; + client_protocols_len = 0; } - - SSL_select_next_proto(out, outlen, - remote_protocols, remote_protocols_len, - our_protocols, our_protocols_len); + if (server_protocols == NULL) { + server_protocols = (unsigned char *)""; + server_protocols_len = 0; + } + + ret = SSL_select_next_proto(out, outlen, + server_protocols, server_protocols_len, + client_protocols, client_protocols_len); + if (alpn && ret != OPENSSL_NPN_NEGOTIATED) + return SSL_TLSEXT_ERR_NOACK; return SSL_TLSEXT_ERR_OK; } @@ -2192,7 +2199,7 @@ void *args) { PySSLContext *ctx = (PySSLContext *)args; - return do_protocol_selection(out, outlen, server, server_len, + return do_protocol_selection(0, out, outlen, server, server_len, ctx->npn_protocols, ctx->npn_protocols_len); } #endif @@ -2244,9 +2251,9 @@ void *args) { PySSLContext *ctx = (PySSLContext *)args; - return do_protocol_selection((unsigned char **)out, outlen, - client_protocols, client_protocols_len, - ctx->alpn_protocols, ctx->alpn_protocols_len); + return do_protocol_selection(1, (unsigned char **)out, outlen, + ctx->alpn_protocols, ctx->alpn_protocols_len, + client_protocols, client_protocols_len); } #endif -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 24 05:05:56 2015 From: python-checkins at python.org (ethan.furman) Date: Sat, 24 Jan 2015 04:05:56 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue20284=3A_Implement_PE?= =?utf-8?q?P461?= Message-ID: <20150124040555.118220.88542@psf.io> https://hg.python.org/cpython/rev/8d802fb6ae32 changeset: 94266:8d802fb6ae32 parent: 94264:eaa38b75cc78 user: Ethan Furman date: Fri Jan 23 20:05:18 2015 -0800 summary: Issue20284: Implement PEP461 files: Doc/library/stdtypes.rst | 191 +++++++ Include/bytesobject.h | 1 + Include/unicodeobject.h | 2 + Lib/test/test_bytes.py | 44 + Lib/test/test_format.py | 385 +++++++++----- Misc/NEWS | 3 + Objects/abstract.c | 5 +- Objects/bytearrayobject.c | 43 +- Objects/bytesobject.c | 657 +++++++++++++++++++++++++- Objects/unicodeobject.c | 12 +- 10 files changed, 1185 insertions(+), 158 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -3057,6 +3057,197 @@ always produces a new object, even if no changes were made. +.. _bytes-formatting: + +``printf``-style Bytes Formatting +---------------------------------- + +.. index:: + single: formatting, bytes (%) + single: formatting, bytearray (%) + single: interpolation, bytes (%) + single: interpolation, bytearray (%) + single: bytes; formatting + single: bytearray; formatting + single: bytes; interpolation + single: bytearray; interpolation + single: printf-style formatting + single: sprintf-style formatting + single: % formatting + single: % interpolation + +.. note:: + + The formatting operations described here exhibit a variety of quirks that + lead to a number of common errors (such as failing to display tuples and + dictionaries correctly). If the value being printed may be a tuple or + dictionary, wrap it in a tuple. + +Bytes objects (``bytes``/``bytearray``) have one unique built-in operation: +the ``%`` operator (modulo). +This is also known as the bytes *formatting* or *interpolation* operator. +Given ``format % values`` (where *format* is a bytes object), ``%`` conversion +specifications in *format* are replaced with zero or more elements of *values*. +The effect is similar to using the :c:func:`sprintf` in the C language. + +If *format* requires a single argument, *values* may be a single non-tuple +object. [5]_ Otherwise, *values* must be a tuple with exactly the number of +items specified by the format bytes object, or a single mapping object (for +example, a dictionary). + +A conversion specifier contains two or more characters and has the following +components, which must occur in this order: + +#. The ``'%'`` character, which marks the start of the specifier. + +#. Mapping key (optional), consisting of a parenthesised sequence of characters + (for example, ``(somename)``). + +#. Conversion flags (optional), which affect the result of some conversion + types. + +#. Minimum field width (optional). If specified as an ``'*'`` (asterisk), the + actual width is read from the next element of the tuple in *values*, and the + object to convert comes after the minimum field width and optional precision. + +#. Precision (optional), given as a ``'.'`` (dot) followed by the precision. If + specified as ``'*'`` (an asterisk), the actual precision is read from the next + element of the tuple in *values*, and the value to convert comes after the + precision. + +#. Length modifier (optional). + +#. Conversion type. + +When the right argument is a dictionary (or other mapping type), then the +formats in the bytes object *must* include a parenthesised mapping key into that +dictionary inserted immediately after the ``'%'`` character. The mapping key +selects the value to be formatted from the mapping. For example: + + >>> print(b'%(language)s has %(number)03d quote types.' % + ... {b'language': b"Python", b"number": 2}) + b'Python has 002 quote types.' + +In this case no ``*`` specifiers may occur in a format (since they require a +sequential parameter list). + +The conversion flag characters are: + ++---------+---------------------------------------------------------------------+ +| Flag | Meaning | ++=========+=====================================================================+ +| ``'#'`` | The value conversion will use the "alternate form" (where defined | +| | below). | ++---------+---------------------------------------------------------------------+ +| ``'0'`` | The conversion will be zero padded for numeric values. | ++---------+---------------------------------------------------------------------+ +| ``'-'`` | The converted value is left adjusted (overrides the ``'0'`` | +| | conversion if both are given). | ++---------+---------------------------------------------------------------------+ +| ``' '`` | (a space) A blank should be left before a positive number (or empty | +| | string) produced by a signed conversion. | ++---------+---------------------------------------------------------------------+ +| ``'+'`` | A sign character (``'+'`` or ``'-'``) will precede the conversion | +| | (overrides a "space" flag). | ++---------+---------------------------------------------------------------------+ + +A length modifier (``h``, ``l``, or ``L``) may be present, but is ignored as it +is not necessary for Python -- so e.g. ``%ld`` is identical to ``%d``. + +The conversion types are: + ++------------+-----------------------------------------------------+-------+ +| Conversion | Meaning | Notes | ++============+=====================================================+=======+ +| ``'d'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'i'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'o'`` | Signed octal value. | \(1) | ++------------+-----------------------------------------------------+-------+ +| ``'u'`` | Obsolete type -- it is identical to ``'d'``. | \(7) | ++------------+-----------------------------------------------------+-------+ +| ``'x'`` | Signed hexadecimal (lowercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'X'`` | Signed hexadecimal (uppercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'e'`` | Floating point exponential format (lowercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'E'`` | Floating point exponential format (uppercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'f'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'F'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'g'`` | Floating point format. Uses lowercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'G'`` | Floating point format. Uses uppercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'c'`` | Single byte (accepts integer or single | | +| | byte objects). | | ++------------+-----------------------------------------------------+-------+ +| ``'b'`` | Bytes (any object that follows the | \(5) | +| | :ref:`buffer protocol ` or has | | +| | :meth:`__bytes__`). | | ++------------+-----------------------------------------------------+-------+ +| ``'s'`` | ``'s'`` is an alias for ``'b'`` and should only | \(6) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'a'`` | Bytes (converts any Python object using | \(5) | +| | ``repr(obj).encode('ascii','backslashreplace)``). | | ++------------+-----------------------------------------------------+-------+ +| ``'%'`` | No argument is converted, results in a ``'%'`` | | +| | character in the result. | | ++------------+-----------------------------------------------------+-------+ + +Notes: + +(1) + The alternate form causes a leading zero (``'0'``) to be inserted between + left-hand padding and the formatting of the number if the leading character + of the result is not already a zero. + +(2) + The alternate form causes a leading ``'0x'`` or ``'0X'`` (depending on whether + the ``'x'`` or ``'X'`` format was used) to be inserted between left-hand padding + and the formatting of the number if the leading character of the result is not + already a zero. + +(3) + The alternate form causes the result to always contain a decimal point, even if + no digits follow it. + + The precision determines the number of digits after the decimal point and + defaults to 6. + +(4) + The alternate form causes the result to always contain a decimal point, and + trailing zeroes are not removed as they would otherwise be. + + The precision determines the number of significant digits before and after the + decimal point and defaults to 6. + +(5) + If precision is ``N``, the output is truncated to ``N`` characters. + +(6) + ``b'%s'`` is deprecated, but will not be removed during the 3.x series. + +(7) + See :pep:`237`. + +.. note:: + + The bytearray version of this method does *not* operate in place - it + always produces a new object, even if no changes were made. + +.. seealso:: :pep:`461`. +.. versionadded:: 3.5 + .. _typememoryview: Memory Views diff --git a/Include/bytesobject.h b/Include/bytesobject.h --- a/Include/bytesobject.h +++ b/Include/bytesobject.h @@ -62,6 +62,7 @@ PyAPI_FUNC(void) PyBytes_ConcatAndDel(PyObject **, PyObject *); #ifndef Py_LIMITED_API PyAPI_FUNC(int) _PyBytes_Resize(PyObject **, Py_ssize_t); +PyAPI_FUNC(PyObject *) _PyBytes_Format(PyObject *, PyObject *); #endif PyAPI_FUNC(PyObject *) PyBytes_DecodeEscape(const char *, Py_ssize_t, const char *, Py_ssize_t, diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -2245,6 +2245,8 @@ Py_UNICODE c ); +PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int); + /* Create a copy of a unicode string ending with a nul character. Return NULL and raise a MemoryError exception on memory allocation failure, otherwise return a new allocated buffer (use PyMem_Free() to free the buffer). */ diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -461,6 +461,28 @@ self.assertEqual(b.rindex(i, 3, 9), 7) self.assertRaises(ValueError, b.rindex, w, 1, 3) + def test_mod(self): + b = b'hello, %b!' + orig = b + b = b % b'world' + self.assertEqual(b, b'hello, world!') + self.assertEqual(orig, b'hello, %b!') + self.assertFalse(b is orig) + b = b'%s / 100 = %d%%' + a = b % (b'seventy-nine', 79) + self.assertEquals(a, b'seventy-nine / 100 = 79%') + + def test_imod(self): + b = b'hello, %b!' + orig = b + b %= b'world' + self.assertEqual(b, b'hello, world!') + self.assertEqual(orig, b'hello, %b!') + self.assertFalse(b is orig) + b = b'%s / 100 = %d%%' + b %= (b'seventy-nine', 79) + self.assertEquals(b, b'seventy-nine / 100 = 79%') + def test_replace(self): b = self.type2test(b'mississippi') self.assertEqual(b.replace(b'i', b'a'), b'massassappa') @@ -990,6 +1012,28 @@ b[8:] = b self.assertEqual(b, bytearray(list(range(8)) + list(range(256)))) + def test_mod(self): + b = bytearray(b'hello, %b!') + orig = b + b = b % b'world' + self.assertEqual(b, b'hello, world!') + self.assertEqual(orig, bytearray(b'hello, %b!')) + self.assertFalse(b is orig) + b = bytearray(b'%s / 100 = %d%%') + a = b % (b'seventy-nine', 79) + self.assertEquals(a, bytearray(b'seventy-nine / 100 = 79%')) + + def test_imod(self): + b = bytearray(b'hello, %b!') + orig = b + b %= b'world' + self.assertEqual(b, b'hello, world!') + self.assertEqual(orig, bytearray(b'hello, %b!')) + self.assertFalse(b is orig) + b = bytearray(b'%s / 100 = %d%%') + b %= (b'seventy-nine', 79) + self.assertEquals(b, bytearray(b'seventy-nine / 100 = 79%')) + def test_iconcat(self): b = bytearray(b"abc") b1 = b diff --git a/Lib/test/test_format.py b/Lib/test/test_format.py --- a/Lib/test/test_format.py +++ b/Lib/test/test_format.py @@ -9,7 +9,7 @@ # test string formatting operator (I am not sure if this is being tested # elsewhere but, surely, some of the given cases are *not* tested because # they crash python) -# test on unicode strings as well +# test on bytes object as well def testformat(formatstr, args, output=None, limit=None, overflowok=False): if verbose: @@ -46,181 +46,209 @@ if verbose: print('yes') +def testcommon(formatstr, args, output=None, limit=None, overflowok=False): + # if formatstr is a str, test str, bytes, and bytearray; + # otherwise, test bytes and bytearry + if isinstance(formatstr, str): + testformat(formatstr, args, output, limit, overflowok) + b_format = formatstr.encode('ascii') + else: + b_format = formatstr + ba_format = bytearray(b_format) + b_args = [] + if not isinstance(args, tuple): + args = (args, ) + b_args = tuple(args) + if output is None: + b_output = ba_output = None + else: + if isinstance(output, str): + b_output = output.encode('ascii') + else: + b_output = output + ba_output = bytearray(b_output) + testformat(b_format, b_args, b_output, limit, overflowok) + testformat(ba_format, b_args, ba_output, limit, overflowok) + class FormatTest(unittest.TestCase): - def test_format(self): - testformat("%.1d", (1,), "1") - testformat("%.*d", (sys.maxsize,1), overflowok=True) # expect overflow - testformat("%.100d", (1,), '00000000000000000000000000000000000000' + + def test_common_format(self): + # test the format identifiers that work the same across + # str, bytes, and bytearrays (integer, float, oct, hex) + testcommon("%.1d", (1,), "1") + testcommon("%.*d", (sys.maxsize,1), overflowok=True) # expect overflow + testcommon("%.100d", (1,), '00000000000000000000000000000000000000' '000000000000000000000000000000000000000000000000000000' '00000001', overflowok=True) - testformat("%#.117x", (1,), '0x00000000000000000000000000000000000' + testcommon("%#.117x", (1,), '0x00000000000000000000000000000000000' '000000000000000000000000000000000000000000000000000000' '0000000000000000000000000001', overflowok=True) - testformat("%#.118x", (1,), '0x00000000000000000000000000000000000' + testcommon("%#.118x", (1,), '0x00000000000000000000000000000000000' '000000000000000000000000000000000000000000000000000000' '00000000000000000000000000001', overflowok=True) - testformat("%f", (1.0,), "1.000000") + testcommon("%f", (1.0,), "1.000000") # these are trying to test the limits of the internal magic-number-length # formatting buffer, if that number changes then these tests are less # effective - testformat("%#.*g", (109, -1.e+49/3.)) - testformat("%#.*g", (110, -1.e+49/3.)) - testformat("%#.*g", (110, -1.e+100/3.)) + testcommon("%#.*g", (109, -1.e+49/3.)) + testcommon("%#.*g", (110, -1.e+49/3.)) + testcommon("%#.*g", (110, -1.e+100/3.)) # test some ridiculously large precision, expect overflow - testformat('%12.*f', (123456, 1.0)) + testcommon('%12.*f', (123456, 1.0)) # check for internal overflow validation on length of precision # these tests should no longer cause overflow in Python # 2.7/3.1 and later. - testformat("%#.*g", (110, -1.e+100/3.)) - testformat("%#.*G", (110, -1.e+100/3.)) - testformat("%#.*f", (110, -1.e+100/3.)) - testformat("%#.*F", (110, -1.e+100/3.)) + testcommon("%#.*g", (110, -1.e+100/3.)) + testcommon("%#.*G", (110, -1.e+100/3.)) + testcommon("%#.*f", (110, -1.e+100/3.)) + testcommon("%#.*F", (110, -1.e+100/3.)) # Formatting of integers. Overflow is not ok - testformat("%x", 10, "a") - testformat("%x", 100000000000, "174876e800") - testformat("%o", 10, "12") - testformat("%o", 100000000000, "1351035564000") - testformat("%d", 10, "10") - testformat("%d", 100000000000, "100000000000") + testcommon("%x", 10, "a") + testcommon("%x", 100000000000, "174876e800") + testcommon("%o", 10, "12") + testcommon("%o", 100000000000, "1351035564000") + testcommon("%d", 10, "10") + testcommon("%d", 100000000000, "100000000000") big = 123456789012345678901234567890 - testformat("%d", big, "123456789012345678901234567890") - testformat("%d", -big, "-123456789012345678901234567890") - testformat("%5d", -big, "-123456789012345678901234567890") - testformat("%31d", -big, "-123456789012345678901234567890") - testformat("%32d", -big, " -123456789012345678901234567890") - testformat("%-32d", -big, "-123456789012345678901234567890 ") - testformat("%032d", -big, "-0123456789012345678901234567890") - testformat("%-032d", -big, "-123456789012345678901234567890 ") - testformat("%034d", -big, "-000123456789012345678901234567890") - testformat("%034d", big, "0000123456789012345678901234567890") - testformat("%0+34d", big, "+000123456789012345678901234567890") - testformat("%+34d", big, " +123456789012345678901234567890") - testformat("%34d", big, " 123456789012345678901234567890") - testformat("%.2d", big, "123456789012345678901234567890") - testformat("%.30d", big, "123456789012345678901234567890") - testformat("%.31d", big, "0123456789012345678901234567890") - testformat("%32.31d", big, " 0123456789012345678901234567890") - testformat("%d", float(big), "123456________________________", 6) + testcommon("%d", big, "123456789012345678901234567890") + testcommon("%d", -big, "-123456789012345678901234567890") + testcommon("%5d", -big, "-123456789012345678901234567890") + testcommon("%31d", -big, "-123456789012345678901234567890") + testcommon("%32d", -big, " -123456789012345678901234567890") + testcommon("%-32d", -big, "-123456789012345678901234567890 ") + testcommon("%032d", -big, "-0123456789012345678901234567890") + testcommon("%-032d", -big, "-123456789012345678901234567890 ") + testcommon("%034d", -big, "-000123456789012345678901234567890") + testcommon("%034d", big, "0000123456789012345678901234567890") + testcommon("%0+34d", big, "+000123456789012345678901234567890") + testcommon("%+34d", big, " +123456789012345678901234567890") + testcommon("%34d", big, " 123456789012345678901234567890") + testcommon("%.2d", big, "123456789012345678901234567890") + testcommon("%.30d", big, "123456789012345678901234567890") + testcommon("%.31d", big, "0123456789012345678901234567890") + testcommon("%32.31d", big, " 0123456789012345678901234567890") + testcommon("%d", float(big), "123456________________________", 6) big = 0x1234567890abcdef12345 # 21 hex digits - testformat("%x", big, "1234567890abcdef12345") - testformat("%x", -big, "-1234567890abcdef12345") - testformat("%5x", -big, "-1234567890abcdef12345") - testformat("%22x", -big, "-1234567890abcdef12345") - testformat("%23x", -big, " -1234567890abcdef12345") - testformat("%-23x", -big, "-1234567890abcdef12345 ") - testformat("%023x", -big, "-01234567890abcdef12345") - testformat("%-023x", -big, "-1234567890abcdef12345 ") - testformat("%025x", -big, "-0001234567890abcdef12345") - testformat("%025x", big, "00001234567890abcdef12345") - testformat("%0+25x", big, "+0001234567890abcdef12345") - testformat("%+25x", big, " +1234567890abcdef12345") - testformat("%25x", big, " 1234567890abcdef12345") - testformat("%.2x", big, "1234567890abcdef12345") - testformat("%.21x", big, "1234567890abcdef12345") - testformat("%.22x", big, "01234567890abcdef12345") - testformat("%23.22x", big, " 01234567890abcdef12345") - testformat("%-23.22x", big, "01234567890abcdef12345 ") - testformat("%X", big, "1234567890ABCDEF12345") - testformat("%#X", big, "0X1234567890ABCDEF12345") - testformat("%#x", big, "0x1234567890abcdef12345") - testformat("%#x", -big, "-0x1234567890abcdef12345") - testformat("%#.23x", -big, "-0x001234567890abcdef12345") - testformat("%#+.23x", big, "+0x001234567890abcdef12345") - testformat("%# .23x", big, " 0x001234567890abcdef12345") - testformat("%#+.23X", big, "+0X001234567890ABCDEF12345") - testformat("%#-+.23X", big, "+0X001234567890ABCDEF12345") - testformat("%#-+26.23X", big, "+0X001234567890ABCDEF12345") - testformat("%#-+27.23X", big, "+0X001234567890ABCDEF12345 ") - testformat("%#+27.23X", big, " +0X001234567890ABCDEF12345") + testcommon("%x", big, "1234567890abcdef12345") + testcommon("%x", -big, "-1234567890abcdef12345") + testcommon("%5x", -big, "-1234567890abcdef12345") + testcommon("%22x", -big, "-1234567890abcdef12345") + testcommon("%23x", -big, " -1234567890abcdef12345") + testcommon("%-23x", -big, "-1234567890abcdef12345 ") + testcommon("%023x", -big, "-01234567890abcdef12345") + testcommon("%-023x", -big, "-1234567890abcdef12345 ") + testcommon("%025x", -big, "-0001234567890abcdef12345") + testcommon("%025x", big, "00001234567890abcdef12345") + testcommon("%0+25x", big, "+0001234567890abcdef12345") + testcommon("%+25x", big, " +1234567890abcdef12345") + testcommon("%25x", big, " 1234567890abcdef12345") + testcommon("%.2x", big, "1234567890abcdef12345") + testcommon("%.21x", big, "1234567890abcdef12345") + testcommon("%.22x", big, "01234567890abcdef12345") + testcommon("%23.22x", big, " 01234567890abcdef12345") + testcommon("%-23.22x", big, "01234567890abcdef12345 ") + testcommon("%X", big, "1234567890ABCDEF12345") + testcommon("%#X", big, "0X1234567890ABCDEF12345") + testcommon("%#x", big, "0x1234567890abcdef12345") + testcommon("%#x", -big, "-0x1234567890abcdef12345") + testcommon("%#.23x", -big, "-0x001234567890abcdef12345") + testcommon("%#+.23x", big, "+0x001234567890abcdef12345") + testcommon("%# .23x", big, " 0x001234567890abcdef12345") + testcommon("%#+.23X", big, "+0X001234567890ABCDEF12345") + testcommon("%#-+.23X", big, "+0X001234567890ABCDEF12345") + testcommon("%#-+26.23X", big, "+0X001234567890ABCDEF12345") + testcommon("%#-+27.23X", big, "+0X001234567890ABCDEF12345 ") + testcommon("%#+27.23X", big, " +0X001234567890ABCDEF12345") # next one gets two leading zeroes from precision, and another from the # 0 flag and the width - testformat("%#+027.23X", big, "+0X0001234567890ABCDEF12345") + testcommon("%#+027.23X", big, "+0X0001234567890ABCDEF12345") # same, except no 0 flag - testformat("%#+27.23X", big, " +0X001234567890ABCDEF12345") + testcommon("%#+27.23X", big, " +0X001234567890ABCDEF12345") big = 0o12345670123456701234567012345670 # 32 octal digits - testformat("%o", big, "12345670123456701234567012345670") - testformat("%o", -big, "-12345670123456701234567012345670") - testformat("%5o", -big, "-12345670123456701234567012345670") - testformat("%33o", -big, "-12345670123456701234567012345670") - testformat("%34o", -big, " -12345670123456701234567012345670") - testformat("%-34o", -big, "-12345670123456701234567012345670 ") - testformat("%034o", -big, "-012345670123456701234567012345670") - testformat("%-034o", -big, "-12345670123456701234567012345670 ") - testformat("%036o", -big, "-00012345670123456701234567012345670") - testformat("%036o", big, "000012345670123456701234567012345670") - testformat("%0+36o", big, "+00012345670123456701234567012345670") - testformat("%+36o", big, " +12345670123456701234567012345670") - testformat("%36o", big, " 12345670123456701234567012345670") - testformat("%.2o", big, "12345670123456701234567012345670") - testformat("%.32o", big, "12345670123456701234567012345670") - testformat("%.33o", big, "012345670123456701234567012345670") - testformat("%34.33o", big, " 012345670123456701234567012345670") - testformat("%-34.33o", big, "012345670123456701234567012345670 ") - testformat("%o", big, "12345670123456701234567012345670") - testformat("%#o", big, "0o12345670123456701234567012345670") - testformat("%#o", -big, "-0o12345670123456701234567012345670") - testformat("%#.34o", -big, "-0o0012345670123456701234567012345670") - testformat("%#+.34o", big, "+0o0012345670123456701234567012345670") - testformat("%# .34o", big, " 0o0012345670123456701234567012345670") - testformat("%#+.34o", big, "+0o0012345670123456701234567012345670") - testformat("%#-+.34o", big, "+0o0012345670123456701234567012345670") - testformat("%#-+37.34o", big, "+0o0012345670123456701234567012345670") - testformat("%#+37.34o", big, "+0o0012345670123456701234567012345670") + testcommon("%o", big, "12345670123456701234567012345670") + testcommon("%o", -big, "-12345670123456701234567012345670") + testcommon("%5o", -big, "-12345670123456701234567012345670") + testcommon("%33o", -big, "-12345670123456701234567012345670") + testcommon("%34o", -big, " -12345670123456701234567012345670") + testcommon("%-34o", -big, "-12345670123456701234567012345670 ") + testcommon("%034o", -big, "-012345670123456701234567012345670") + testcommon("%-034o", -big, "-12345670123456701234567012345670 ") + testcommon("%036o", -big, "-00012345670123456701234567012345670") + testcommon("%036o", big, "000012345670123456701234567012345670") + testcommon("%0+36o", big, "+00012345670123456701234567012345670") + testcommon("%+36o", big, " +12345670123456701234567012345670") + testcommon("%36o", big, " 12345670123456701234567012345670") + testcommon("%.2o", big, "12345670123456701234567012345670") + testcommon("%.32o", big, "12345670123456701234567012345670") + testcommon("%.33o", big, "012345670123456701234567012345670") + testcommon("%34.33o", big, " 012345670123456701234567012345670") + testcommon("%-34.33o", big, "012345670123456701234567012345670 ") + testcommon("%o", big, "12345670123456701234567012345670") + testcommon("%#o", big, "0o12345670123456701234567012345670") + testcommon("%#o", -big, "-0o12345670123456701234567012345670") + testcommon("%#.34o", -big, "-0o0012345670123456701234567012345670") + testcommon("%#+.34o", big, "+0o0012345670123456701234567012345670") + testcommon("%# .34o", big, " 0o0012345670123456701234567012345670") + testcommon("%#+.34o", big, "+0o0012345670123456701234567012345670") + testcommon("%#-+.34o", big, "+0o0012345670123456701234567012345670") + testcommon("%#-+37.34o", big, "+0o0012345670123456701234567012345670") + testcommon("%#+37.34o", big, "+0o0012345670123456701234567012345670") # next one gets one leading zero from precision - testformat("%.33o", big, "012345670123456701234567012345670") + testcommon("%.33o", big, "012345670123456701234567012345670") # base marker shouldn't change that, since "0" is redundant - testformat("%#.33o", big, "0o012345670123456701234567012345670") + testcommon("%#.33o", big, "0o012345670123456701234567012345670") # but reduce precision, and base marker should add a zero - testformat("%#.32o", big, "0o12345670123456701234567012345670") + testcommon("%#.32o", big, "0o12345670123456701234567012345670") # one leading zero from precision, and another from "0" flag & width - testformat("%034.33o", big, "0012345670123456701234567012345670") + testcommon("%034.33o", big, "0012345670123456701234567012345670") # base marker shouldn't change that - testformat("%0#34.33o", big, "0o012345670123456701234567012345670") + testcommon("%0#34.33o", big, "0o012345670123456701234567012345670") # Some small ints, in both Python int and flavors). - testformat("%d", 42, "42") - testformat("%d", -42, "-42") - testformat("%d", 42, "42") - testformat("%d", -42, "-42") - testformat("%d", 42.0, "42") - testformat("%#x", 1, "0x1") - testformat("%#x", 1, "0x1") - testformat("%#X", 1, "0X1") - testformat("%#X", 1, "0X1") - testformat("%#o", 1, "0o1") - testformat("%#o", 1, "0o1") - testformat("%#o", 0, "0o0") - testformat("%#o", 0, "0o0") - testformat("%o", 0, "0") - testformat("%o", 0, "0") - testformat("%d", 0, "0") - testformat("%d", 0, "0") - testformat("%#x", 0, "0x0") - testformat("%#x", 0, "0x0") - testformat("%#X", 0, "0X0") - testformat("%#X", 0, "0X0") - testformat("%x", 0x42, "42") - testformat("%x", -0x42, "-42") - testformat("%x", 0x42, "42") - testformat("%x", -0x42, "-42") - testformat("%o", 0o42, "42") - testformat("%o", -0o42, "-42") - testformat("%o", 0o42, "42") - testformat("%o", -0o42, "-42") + testcommon("%d", 42, "42") + testcommon("%d", -42, "-42") + testcommon("%d", 42, "42") + testcommon("%d", -42, "-42") + testcommon("%d", 42.0, "42") + testcommon("%#x", 1, "0x1") + testcommon("%#x", 1, "0x1") + testcommon("%#X", 1, "0X1") + testcommon("%#X", 1, "0X1") + testcommon("%#o", 1, "0o1") + testcommon("%#o", 1, "0o1") + testcommon("%#o", 0, "0o0") + testcommon("%#o", 0, "0o0") + testcommon("%o", 0, "0") + testcommon("%o", 0, "0") + testcommon("%d", 0, "0") + testcommon("%d", 0, "0") + testcommon("%#x", 0, "0x0") + testcommon("%#x", 0, "0x0") + testcommon("%#X", 0, "0X0") + testcommon("%#X", 0, "0X0") + testcommon("%x", 0x42, "42") + testcommon("%x", -0x42, "-42") + testcommon("%x", 0x42, "42") + testcommon("%x", -0x42, "-42") + testcommon("%o", 0o42, "42") + testcommon("%o", -0o42, "-42") + testcommon("%o", 0o42, "42") + testcommon("%o", -0o42, "-42") + # alternate float formatting + testcommon('%g', 1.1, '1.1') + testcommon('%#g', 1.1, '1.10000') + + def test_str_format(self): testformat("%r", "\u0378", "'\\u0378'") # non printable testformat("%a", "\u0378", "'\\u0378'") # non printable testformat("%r", "\u0374", "'\u0374'") # printable testformat("%a", "\u0374", "'\\u0374'") # printable - # alternate float formatting - testformat('%g', 1.1, '1.1') - testformat('%#g', 1.1, '1.10000') - - # Test exception for unknown format characters + # Test exception for unknown format characters, etc. if verbose: print('Testing exceptions') def test_exc(formatstr, args, exception, excmsg): @@ -247,8 +275,83 @@ test_exc('%g', '1', TypeError, "a float is required") test_exc('no format', '1', TypeError, "not all arguments converted during string formatting") - test_exc('no format', '1', TypeError, - "not all arguments converted during string formatting") + + if maxsize == 2**31-1: + # crashes 2.2.1 and earlier: + try: + "%*d"%(maxsize, -127) + except MemoryError: + pass + else: + raise TestFailed('"%*d"%(maxsize, -127) should fail') + + def test_bytes_and_bytearray_format(self): + # %c will insert a single byte, either from an int in range(256), or + # from a bytes argument of length 1, not from a str. + testcommon(b"%c", 7, b"\x07") + testcommon(b"%c", b"Z", b"Z") + testcommon(b"%c", bytearray(b"Z"), b"Z") + # %b will insert a series of bytes, either from a type that supports + # the Py_buffer protocol, or something that has a __bytes__ method + class FakeBytes(object): + def __bytes__(self): + return b'123' + fb = FakeBytes() + testcommon(b"%b", b"abc", b"abc") + testcommon(b"%b", bytearray(b"def"), b"def") + testcommon(b"%b", fb, b"123") + # # %s is an alias for %b -- should only be used for Py2/3 code + testcommon(b"%s", b"abc", b"abc") + testcommon(b"%s", bytearray(b"def"), b"def") + testcommon(b"%s", fb, b"123") + # %a will give the equivalent of + # repr(some_obj).encode('ascii', 'backslashreplace') + testcommon(b"%a", 3.14, b"3.14") + testcommon(b"%a", b"ghi", b"b'ghi'") + testcommon(b"%a", "jkl", b"'jkl'") + testcommon(b"%a", "\u0544", b"'\\u0544'") + + # Test exception for unknown format characters, etc. + if verbose: + print('Testing exceptions') + def test_exc(formatstr, args, exception, excmsg): + try: + testformat(formatstr, args) + except exception as exc: + if str(exc) == excmsg: + if verbose: + print("yes") + else: + if verbose: print('no') + print('Unexpected ', exception, ':', repr(str(exc))) + except: + if verbose: print('no') + print('Unexpected exception') + raise + else: + raise TestFailed('did not get expected exception: %s' % excmsg) + test_exc(b'%d', '1', TypeError, + "%d format: a number is required, not str") + test_exc(b'%d', b'1', TypeError, + "%d format: a number is required, not bytes") + test_exc(b'%g', '1', TypeError, "float argument required, not str") + test_exc(b'%g', b'1', TypeError, "float argument required, not bytes") + test_exc(b'no format', 7, TypeError, + "not all arguments converted during bytes formatting") + test_exc(b'no format', b'1', TypeError, + "not all arguments converted during bytes formatting") + test_exc(b'no format', bytearray(b'1'), TypeError, + "not all arguments converted during bytes formatting") + test_exc(b"%c", 256, TypeError, + "%c requires an integer in range(256) or a single byte") + test_exc(b"%c", b"Za", TypeError, + "%c requires an integer in range(256) or a single byte") + test_exc(b"%c", "Yb", TypeError, + "%c requires an integer in range(256) or a single byte") + test_exc(b"%b", "Xc", TypeError, + "%b requires bytes, or an object that implements __bytes__, not 'str'") + test_exc(b"%s", "Wd", TypeError, + "%b requires bytes, or an object that implements __bytes__, not 'str'") if maxsize == 2**31-1: # crashes 2.2.1 and earlier: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -14,6 +14,9 @@ atomic memory access if available. Patch written by Vitor de Lima and Gustavo Temple. +- Issue #20284: %-interpolation (aka printf) formatting added for bytes and + bytearray. + - Issue #23048: Fix jumping out of an infinite while loop in the pdb. - Issue #20335: bytes constructor now raises TypeError when encoding or errors diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -686,8 +686,9 @@ Py_DECREF(meth); if (result && !PyUnicode_Check(result)) { - PyErr_SetString(PyExc_TypeError, - "__format__ method did not return string"); + PyErr_Format(PyExc_TypeError, + "__format__ must return a str, not %.200s", + Py_TYPE(result)->tp_name); Py_DECREF(result); result = NULL; goto done; diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -4,6 +4,7 @@ #include "Python.h" #include "structmember.h" #include "bytes_methods.h" +#include "bytesobject.h" /*[clinic input] class bytearray "PyByteArrayObject *" "&PyByteArray_Type" @@ -294,6 +295,31 @@ return (PyObject *)result; } +static PyObject * +bytearray_format(PyByteArrayObject *self, PyObject *args) +{ + PyObject *bytes_in, *bytes_out, *res; + char *bytestring; + + if (self == NULL || !PyByteArray_Check(self) || args == NULL) { + PyErr_BadInternalCall(); + return NULL; + } + bytestring = PyByteArray_AS_STRING(self); + bytes_in = PyBytes_FromString(bytestring); + if (bytes_in == NULL) + return NULL; + bytes_out = _PyBytes_Format(bytes_in, args); + Py_DECREF(bytes_in); + if (bytes_out == NULL) + return NULL; + res = PyByteArray_FromObject(bytes_out); + Py_DECREF(bytes_out); + if (res == NULL) + return NULL; + return res; +} + /* Functions stuffed into the type object */ static Py_ssize_t @@ -3723,6 +3749,21 @@ {NULL} }; +static PyObject * +bytearray_mod(PyObject *v, PyObject *w) +{ + if (!PyByteArray_Check(v)) + Py_RETURN_NOTIMPLEMENTED; + return bytearray_format((PyByteArrayObject *)v, w); +} + +static PyNumberMethods bytearray_as_number = { + 0, /*nb_add*/ + 0, /*nb_subtract*/ + 0, /*nb_multiply*/ + bytearray_mod, /*nb_remainder*/ +}; + PyDoc_STRVAR(bytearray_doc, "bytearray(iterable_of_ints) -> bytearray\n\ bytearray(string, encoding[, errors]) -> bytearray\n\ @@ -3751,7 +3792,7 @@ 0, /* tp_setattr */ 0, /* tp_reserved */ (reprfunc)bytearray_repr, /* tp_repr */ - 0, /* tp_as_number */ + &bytearray_as_number, /* tp_as_number */ &bytearray_as_sequence, /* tp_as_sequence */ &bytearray_as_mapping, /* tp_as_mapping */ 0, /* tp_hash */ diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -400,6 +400,634 @@ return ret; } +/* Helpers for formatstring */ + +Py_LOCAL_INLINE(PyObject *) +getnextarg(PyObject *args, Py_ssize_t arglen, Py_ssize_t *p_argidx) +{ + Py_ssize_t argidx = *p_argidx; + if (argidx < arglen) { + (*p_argidx)++; + if (arglen < 0) + return args; + else + return PyTuple_GetItem(args, argidx); + } + PyErr_SetString(PyExc_TypeError, + "not enough arguments for format string"); + return NULL; +} + +/* Format codes + * F_LJUST '-' + * F_SIGN '+' + * F_BLANK ' ' + * F_ALT '#' + * F_ZERO '0' + */ +#define F_LJUST (1<<0) +#define F_SIGN (1<<1) +#define F_BLANK (1<<2) +#define F_ALT (1<<3) +#define F_ZERO (1<<4) + +/* Returns a new reference to a PyBytes object, or NULL on failure. */ + +static PyObject * +formatfloat(PyObject *v, int flags, int prec, int type) +{ + char *p; + PyObject *result; + double x; + + x = PyFloat_AsDouble(v); + if (x == -1.0 && PyErr_Occurred()) { + PyErr_Format(PyExc_TypeError, "float argument required, " + "not %.200s", Py_TYPE(v)->tp_name); + return NULL; + } + + if (prec < 0) + prec = 6; + + p = PyOS_double_to_string(x, type, prec, + (flags & F_ALT) ? Py_DTSF_ALT : 0, NULL); + + if (p == NULL) + return NULL; + result = PyBytes_FromStringAndSize(p, strlen(p)); + PyMem_Free(p); + return result; +} + +/* format_long emulates the format codes d, u, o, x and X, and + * the F_ALT flag, for Python's long (unbounded) ints. It's not used for + * Python's regular ints. + * Return value: a new PyBytes*, or NULL if error. + * . *pbuf is set to point into it, + * *plen set to the # of chars following that. + * Caller must decref it when done using pbuf. + * The string starting at *pbuf is of the form + * "-"? ("0x" | "0X")? digit+ + * "0x"/"0X" are present only for x and X conversions, with F_ALT + * set in flags. The case of hex digits will be correct, + * There will be at least prec digits, zero-filled on the left if + * necessary to get that many. + * val object to be converted + * flags bitmask of format flags; only F_ALT is looked at + * prec minimum number of digits; 0-fill on left if needed + * type a character in [duoxX]; u acts the same as d + * + * CAUTION: o, x and X conversions on regular ints can never + * produce a '-' sign, but can for Python's unbounded ints. + */ + +static PyObject * +format_long(PyObject *val, int flags, int prec, int type, + char **pbuf, int *plen) +{ + PyObject *s; + PyObject *result = NULL; + + s = _PyUnicode_FormatLong(val, flags & F_ALT, prec, type); + if (!s) + return NULL; + result = _PyUnicode_AsASCIIString(s, "strict"); + Py_DECREF(s); + if (!result) + return NULL; + *pbuf = PyBytes_AS_STRING(result); + *plen = PyBytes_GET_SIZE(result); + return result; +} + +Py_LOCAL_INLINE(int) +formatchar(char *buf, size_t buflen, PyObject *v) +{ + PyObject *w = NULL; + /* convert bytearray to bytes */ + if (PyByteArray_Check(v)) { + w = PyBytes_FromObject(v); + if (w == NULL) + goto error; + v = w; + } + /* presume that the buffer is at least 2 characters long */ + if (PyBytes_Check(v)) { + if (!PyArg_Parse(v, "c;%c requires an integer in range(256) or a single byte", &buf[0])) + goto error; + } + else { + long ival = PyLong_AsLong(v); + if (ival == -1 && PyErr_Occurred()) { + PyErr_SetString(PyExc_TypeError, + "%c requires an integer in range(256) or a single byte"); + goto error; + } + if (ival < 0 || ival > 255) { + PyErr_SetString(PyExc_TypeError, + "%c requires an integer in range(256) or a single byte"); + goto error; + } + buf[0] = ival; + } + Py_XDECREF(w); + buf[1] = '\0'; + return 1; + + error: + Py_XDECREF(w); + return -1; +} + +static PyObject * +format_obj(PyObject *v) +{ + PyObject *result = NULL, *w = NULL; + PyObject *func; + _Py_IDENTIFIER(__bytes__); + /* convert bytearray to bytes */ + if (PyByteArray_Check(v)) { + w = PyBytes_FromObject(v); + if (w == NULL) + return NULL; + v = w; + } + /* is it a bytes object? */ + if (PyBytes_Check(v)) { + result = v; + Py_INCREF(v); + Py_XDECREF(w); + return result; + } + /* does it support __bytes__? */ + func = _PyObject_LookupSpecial(v, &PyId___bytes__); + if (func != NULL) { + result = PyObject_CallFunctionObjArgs(func, NULL); + Py_DECREF(func); + if (result == NULL) + return NULL; + if (!PyBytes_Check(result)) { + PyErr_Format(PyExc_TypeError, + "__bytes__ returned non-bytes (type %.200s)", + Py_TYPE(result)->tp_name); + Py_DECREF(result); + return NULL; + } + return result; + } + PyErr_Format(PyExc_TypeError, + "%%b requires bytes, or an object that implements __bytes__, not '%.100s'", + Py_TYPE(v)->tp_name); + return NULL; +} + +/* fmt%(v1,v2,...) is roughly equivalent to sprintf(fmt, v1, v2, ...) + + FORMATBUFLEN is the length of the buffer in which the ints & + chars are formatted. XXX This is a magic number. Each formatting + routine does bounds checking to ensure no overflow, but a better + solution may be to malloc a buffer of appropriate size for each + format. For now, the current solution is sufficient. +*/ +#define FORMATBUFLEN (size_t)120 + +PyObject * +_PyBytes_Format(PyObject *format, PyObject *args) +{ + char *fmt, *res; + Py_ssize_t arglen, argidx; + Py_ssize_t reslen, rescnt, fmtcnt; + int args_owned = 0; + PyObject *result; + PyObject *repr; + PyObject *dict = NULL; + if (format == NULL || !PyBytes_Check(format) || args == NULL) { + PyErr_BadInternalCall(); + return NULL; + } + fmt = PyBytes_AS_STRING(format); + fmtcnt = PyBytes_GET_SIZE(format); + reslen = rescnt = fmtcnt + 100; + result = PyBytes_FromStringAndSize((char *)NULL, reslen); + if (result == NULL) + return NULL; + res = PyBytes_AsString(result); + if (PyTuple_Check(args)) { + arglen = PyTuple_GET_SIZE(args); + argidx = 0; + } + else { + arglen = -1; + argidx = -2; + } + if (Py_TYPE(args)->tp_as_mapping && Py_TYPE(args)->tp_as_mapping->mp_subscript && + !PyTuple_Check(args) && !PyBytes_Check(args) && !PyUnicode_Check(args) && + !PyByteArray_Check(args)) { + dict = args; + } + while (--fmtcnt >= 0) { + if (*fmt != '%') { + if (--rescnt < 0) { + rescnt = fmtcnt + 100; + reslen += rescnt; + if (_PyBytes_Resize(&result, reslen)) + return NULL; + res = PyBytes_AS_STRING(result) + + reslen - rescnt; + --rescnt; + } + *res++ = *fmt++; + } + else { + /* Got a format specifier */ + int flags = 0; + Py_ssize_t width = -1; + int prec = -1; + int c = '\0'; + int fill; + int isnumok; + PyObject *v = NULL; + PyObject *temp = NULL; + Py_buffer buf; + char *pbuf; + int sign; + Py_ssize_t len; + char formatbuf[FORMATBUFLEN]; + /* For format{int,char}() */ + + buf.obj = NULL; + fmt++; + if (*fmt == '(') { + char *keystart; + Py_ssize_t keylen; + PyObject *key; + int pcount = 1; + + if (dict == NULL) { + PyErr_SetString(PyExc_TypeError, + "format requires a mapping"); + goto error; + } + ++fmt; + --fmtcnt; + keystart = fmt; + /* Skip over balanced parentheses */ + while (pcount > 0 && --fmtcnt >= 0) { + if (*fmt == ')') + --pcount; + else if (*fmt == '(') + ++pcount; + fmt++; + } + keylen = fmt - keystart - 1; + if (fmtcnt < 0 || pcount > 0) { + PyErr_SetString(PyExc_ValueError, + "incomplete format key"); + goto error; + } + key = PyBytes_FromStringAndSize(keystart, + keylen); + if (key == NULL) + goto error; + if (args_owned) { + Py_DECREF(args); + args_owned = 0; + } + args = PyObject_GetItem(dict, key); + Py_DECREF(key); + if (args == NULL) { + goto error; + } + args_owned = 1; + arglen = -1; + argidx = -2; + } + while (--fmtcnt >= 0) { + switch (c = *fmt++) { + case '-': flags |= F_LJUST; continue; + case '+': flags |= F_SIGN; continue; + case ' ': flags |= F_BLANK; continue; + case '#': flags |= F_ALT; continue; + case '0': flags |= F_ZERO; continue; + } + break; + } + if (c == '*') { + v = getnextarg(args, arglen, &argidx); + if (v == NULL) + goto error; + if (!PyLong_Check(v)) { + PyErr_SetString(PyExc_TypeError, + "* wants int"); + goto error; + } + width = PyLong_AsSsize_t(v); + if (width == -1 && PyErr_Occurred()) + goto error; + if (width < 0) { + flags |= F_LJUST; + width = -width; + } + if (--fmtcnt >= 0) + c = *fmt++; + } + else if (c >= 0 && isdigit(c)) { + width = c - '0'; + while (--fmtcnt >= 0) { + c = Py_CHARMASK(*fmt++); + if (!isdigit(c)) + break; + if (width > (PY_SSIZE_T_MAX - ((int)c - '0')) / 10) { + PyErr_SetString( + PyExc_ValueError, + "width too big"); + goto error; + } + width = width*10 + (c - '0'); + } + } + if (c == '.') { + prec = 0; + if (--fmtcnt >= 0) + c = *fmt++; + if (c == '*') { + v = getnextarg(args, arglen, &argidx); + if (v == NULL) + goto error; + if (!PyLong_Check(v)) { + PyErr_SetString( + PyExc_TypeError, + "* wants int"); + goto error; + } + prec = PyLong_AsSsize_t(v); + if (prec == -1 && PyErr_Occurred()) + goto error; + if (prec < 0) + prec = 0; + if (--fmtcnt >= 0) + c = *fmt++; + } + else if (c >= 0 && isdigit(c)) { + prec = c - '0'; + while (--fmtcnt >= 0) { + c = Py_CHARMASK(*fmt++); + if (!isdigit(c)) + break; + if (prec > (INT_MAX - ((int)c - '0')) / 10) { + PyErr_SetString( + PyExc_ValueError, + "prec too big"); + goto error; + } + prec = prec*10 + (c - '0'); + } + } + } /* prec */ + if (fmtcnt >= 0) { + if (c == 'h' || c == 'l' || c == 'L') { + if (--fmtcnt >= 0) + c = *fmt++; + } + } + if (fmtcnt < 0) { + PyErr_SetString(PyExc_ValueError, + "incomplete format"); + goto error; + } + if (c != '%') { + v = getnextarg(args, arglen, &argidx); + if (v == NULL) + goto error; + } + sign = 0; + fill = ' '; + switch (c) { + case '%': + pbuf = "%"; + len = 1; + break; + case 'a': + temp = PyObject_Repr(v); + if (temp == NULL) + goto error; + repr = PyUnicode_AsEncodedObject(temp, "ascii", "backslashreplace"); + if (repr == NULL) { + Py_DECREF(temp); + goto error; + } + if (_getbuffer(repr, &buf) < 0) { + temp = format_obj(repr); + if (temp == NULL) { + Py_DECREF(repr); + goto error; + } + Py_DECREF(repr); + repr = temp; + } + pbuf = PyBytes_AS_STRING(repr); + len = PyBytes_GET_SIZE(repr); + Py_DECREF(repr); + if (prec >= 0 && len > prec) + len = prec; + break; + case 's': + // %s is only for 2/3 code; 3 only code should use %b + case 'b': + temp = format_obj(v); + if (temp == NULL) + goto error; + pbuf = PyBytes_AS_STRING(temp); + len = PyBytes_GET_SIZE(temp); + if (prec >= 0 && len > prec) + len = prec; + break; + case 'i': + case 'd': + case 'u': + case 'o': + case 'x': + case 'X': + if (c == 'i') + c = 'd'; + isnumok = 0; + if (PyNumber_Check(v)) { + PyObject *iobj=NULL; + + if ((PyLong_Check(v))) { + iobj = v; + Py_INCREF(iobj); + } + else { + iobj = PyNumber_Long(v); + } + if (iobj!=NULL) { + if (PyLong_Check(iobj)) { + int ilen; + + isnumok = 1; + temp = format_long(iobj, flags, prec, c, + &pbuf, &ilen); + Py_DECREF(iobj); + len = ilen; + if (!temp) + goto error; + sign = 1; + } + else { + Py_DECREF(iobj); + } + } + } + if (!isnumok) { + PyErr_Format(PyExc_TypeError, + "%%%c format: a number is required, " + "not %.200s", c, Py_TYPE(v)->tp_name); + goto error; + } + if (flags & F_ZERO) + fill = '0'; + break; + case 'e': + case 'E': + case 'f': + case 'F': + case 'g': + case 'G': + temp = formatfloat(v, flags, prec, c); + if (temp == NULL) + goto error; + pbuf = PyBytes_AS_STRING(temp); + len = PyBytes_GET_SIZE(temp); + sign = 1; + if (flags & F_ZERO) + fill = '0'; + break; + case 'c': + pbuf = formatbuf; + len = formatchar(pbuf, sizeof(formatbuf), v); + if (len < 0) + goto error; + break; + default: + PyErr_Format(PyExc_ValueError, + "unsupported format character '%c' (0x%x) " + "at index %zd", + c, c, + (Py_ssize_t)(fmt - 1 - + PyBytes_AsString(format))); + goto error; + } + if (sign) { + if (*pbuf == '-' || *pbuf == '+') { + sign = *pbuf++; + len--; + } + else if (flags & F_SIGN) + sign = '+'; + else if (flags & F_BLANK) + sign = ' '; + else + sign = 0; + } + if (width < len) + width = len; + if (rescnt - (sign != 0) < width) { + reslen -= rescnt; + rescnt = width + fmtcnt + 100; + reslen += rescnt; + if (reslen < 0) { + Py_DECREF(result); + PyBuffer_Release(&buf); + Py_XDECREF(temp); + return PyErr_NoMemory(); + } + if (_PyBytes_Resize(&result, reslen)) { + PyBuffer_Release(&buf); + Py_XDECREF(temp); + return NULL; + } + res = PyBytes_AS_STRING(result) + + reslen - rescnt; + } + if (sign) { + if (fill != ' ') + *res++ = sign; + rescnt--; + if (width > len) + width--; + } + if ((flags & F_ALT) && (c == 'x' || c == 'X')) { + assert(pbuf[0] == '0'); + assert(pbuf[1] == c); + if (fill != ' ') { + *res++ = *pbuf++; + *res++ = *pbuf++; + } + rescnt -= 2; + width -= 2; + if (width < 0) + width = 0; + len -= 2; + } + if (width > len && !(flags & F_LJUST)) { + do { + --rescnt; + *res++ = fill; + } while (--width > len); + } + if (fill == ' ') { + if (sign) + *res++ = sign; + if ((flags & F_ALT) && + (c == 'x' || c == 'X')) { + assert(pbuf[0] == '0'); + assert(pbuf[1] == c); + *res++ = *pbuf++; + *res++ = *pbuf++; + } + } + Py_MEMCPY(res, pbuf, len); + res += len; + rescnt -= len; + while (--width >= len) { + --rescnt; + *res++ = ' '; + } + if (dict && (argidx < arglen) && c != '%') { + PyErr_SetString(PyExc_TypeError, + "not all arguments converted during bytes formatting"); + PyBuffer_Release(&buf); + Py_XDECREF(temp); + goto error; + } + PyBuffer_Release(&buf); + Py_XDECREF(temp); + } /* '%' */ + } /* until end */ + if (argidx < arglen && !dict) { + PyErr_SetString(PyExc_TypeError, + "not all arguments converted during bytes formatting"); + goto error; + } + if (args_owned) { + Py_DECREF(args); + } + if (_PyBytes_Resize(&result, reslen - rescnt)) + return NULL; + return result; + + error: + Py_DECREF(result); + if (args_owned) { + Py_DECREF(args); + } + return NULL; +} + +/* =-= */ + static void bytes_dealloc(PyObject *op) { @@ -2996,6 +3624,21 @@ }; static PyObject * +bytes_mod(PyObject *v, PyObject *w) +{ + if (!PyBytes_Check(v)) + Py_RETURN_NOTIMPLEMENTED; + return _PyBytes_Format(v, w); +} + +static PyNumberMethods bytes_as_number = { + 0, /*nb_add*/ + 0, /*nb_subtract*/ + 0, /*nb_multiply*/ + bytes_mod, /*nb_remainder*/ +}; + +static PyObject * str_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds); static PyObject * @@ -3286,7 +3929,7 @@ 0, /* tp_setattr */ 0, /* tp_reserved */ (reprfunc)bytes_repr, /* tp_repr */ - 0, /* tp_as_number */ + &bytes_as_number, /* tp_as_number */ &bytes_as_sequence, /* tp_as_sequence */ &bytes_as_mapping, /* tp_as_mapping */ (hashfunc)bytes_hash, /* tp_hash */ @@ -3377,14 +4020,14 @@ } -/* The following function breaks the notion that strings are immutable: - it changes the size of a string. We get away with this only if there +/* The following function breaks the notion that bytes are immutable: + it changes the size of a bytes object. We get away with this only if there is only one module referencing the object. You can also think of it - as creating a new string object and destroying the old one, only - more efficiently. In any case, don't use this if the string may + as creating a new bytes object and destroying the old one, only + more efficiently. In any case, don't use this if the bytes object may already be known to some other part of the code... - Note that if there's not enough memory to resize the string, the original - string object at *pv is deallocated, *pv is set to NULL, an "out of + Note that if there's not enough memory to resize the bytes object, the + original bytes object at *pv is deallocated, *pv is set to NULL, an "out of memory" exception is set, and -1 is returned. Else (on success) 0 is returned, and the value in *pv may or may not be the same as on input. As always, an extra byte is allocated for a trailing \0 byte (newsize diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13893,8 +13893,8 @@ * CAUTION: o, x and X conversions on regular ints can never * produce a '-' sign, but can for Python's unbounded ints. */ -static PyObject* -formatlong(PyObject *val, struct unicode_format_arg_t *arg) +PyObject * +_PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type) { PyObject *result = NULL; char *buf; @@ -13904,8 +13904,6 @@ Py_ssize_t llen; int numdigits; /* len == numnondigits + numdigits */ int numnondigits = 0; - int prec = arg->prec; - int type = arg->ch; /* Avoid exceeding SSIZE_T_MAX */ if (prec > INT_MAX-3) { @@ -13954,7 +13952,7 @@ if (llen > INT_MAX) { Py_DECREF(result); PyErr_SetString(PyExc_ValueError, - "string too large in _PyBytes_FormatLong"); + "string too large in _PyUnicode_FormatLong"); return NULL; } len = (int)llen; @@ -13964,7 +13962,7 @@ assert(numdigits > 0); /* Get rid of base marker unless F_ALT */ - if (((arg->flags & F_ALT) == 0 && + if (((alt) == 0 && (type == 'o' || type == 'x' || type == 'X'))) { assert(buf[sign] == '0'); assert(buf[sign+1] == 'x' || buf[sign+1] == 'X' || @@ -14099,7 +14097,7 @@ return 1; } - res = formatlong(iobj, arg); + res = _PyUnicode_FormatLong(iobj, arg->flags & F_ALT, arg->prec, type); Py_DECREF(iobj); if (res == NULL) return -1; -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Sat Jan 24 10:00:49 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 24 Jan 2015 10:00:49 +0100 Subject: [Python-checkins] Daily reference leaks (eaa38b75cc78): sum=3 Message-ID: results for eaa38b75cc78 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogUryxRQ', '-x'] From python-checkins at python.org Sat Jan 24 17:54:06 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 24 Jan 2015 16:54:06 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2323253=3A_Delay-l?= =?utf-8?q?oad_ShellExecute?= Message-ID: <20150124165355.84263.87428@psf.io> https://hg.python.org/cpython/rev/5bff604a864e changeset: 94267:5bff604a864e user: Steve Dower date: Sat Jan 24 08:18:24 2015 -0800 summary: Closes #23253: Delay-load ShellExecute files: Doc/library/os.rst | 4 ++ Misc/NEWS | 3 + Modules/posixmodule.c | 47 ++++++++++++++++++++++++++++-- 3 files changed, 50 insertions(+), 4 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3024,6 +3024,10 @@ doesn't work if it is. Use the :func:`os.path.normpath` function to ensure that the path is properly encoded for Win32. + To reduce interpreter startup overhead, the Win32 :c:func:`ShellExecute` + function is not resolved until this function is first called. If the function + cannot be resolved, :exc:`NotImplementedError` will be raised. + Availability: Windows. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #23253: Delay-load ShellExecute[AW] in os.startfile for reduced + startup overhead on Windows. + - Issue #22038: pyatomic.h now uses stdatomic.h or GCC built-in functions for atomic memory access if available. Patch written by Vitor de Lima and Gustavo Temple. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -15128,6 +15128,37 @@ an absolute path, make sure the first character is not a slash (\"/\");\n\ the underlying Win32 ShellExecute function doesn't work if it is."); +/* Grab ShellExecute dynamically from shell32 */ +static int has_ShellExecute = -1; +static HINSTANCE (CALLBACK *Py_ShellExecuteA)(HWND, LPCSTR, LPCSTR, LPCSTR, + LPCSTR, INT); +static HINSTANCE (CALLBACK *Py_ShellExecuteW)(HWND, LPCWSTR, LPCWSTR, LPCWSTR, + LPCWSTR, INT); +static int +check_ShellExecute() +{ + HINSTANCE hShell32; + + /* only recheck */ + if (-1 == has_ShellExecute) { + Py_BEGIN_ALLOW_THREADS + hShell32 = LoadLibraryW(L"SHELL32"); + Py_END_ALLOW_THREADS + if (hShell32) { + *(FARPROC*)&Py_ShellExecuteA = GetProcAddress(hShell32, + "ShellExecuteA"); + *(FARPROC*)&Py_ShellExecuteW = GetProcAddress(hShell32, + "ShellExecuteW"); + has_ShellExecute = Py_ShellExecuteA && + Py_ShellExecuteW; + } else { + has_ShellExecute = 0; + } + } + return has_ShellExecute; +} + + static PyObject * win32_startfile(PyObject *self, PyObject *args) { @@ -15138,6 +15169,14 @@ HINSTANCE rc; PyObject *unipath, *uoperation = NULL; + + if(!check_ShellExecute()) { + /* If the OS doesn't have ShellExecute, return a + NotImplementedError. */ + return PyErr_Format(PyExc_NotImplementedError, + "startfile not available on this platform"); + } + if (!PyArg_ParseTuple(args, "U|s:startfile", &unipath, &operation)) { PyErr_Clear(); @@ -15166,8 +15205,8 @@ woperation = NULL; Py_BEGIN_ALLOW_THREADS - rc = ShellExecuteW((HWND)0, woperation, wpath, - NULL, NULL, SW_SHOWNORMAL); + rc = Py_ShellExecuteW((HWND)0, woperation, wpath, + NULL, NULL, SW_SHOWNORMAL); Py_END_ALLOW_THREADS Py_XDECREF(uoperation); @@ -15189,8 +15228,8 @@ } filepath = PyBytes_AsString(ofilepath); Py_BEGIN_ALLOW_THREADS - rc = ShellExecute((HWND)0, operation, filepath, - NULL, NULL, SW_SHOWNORMAL); + rc = Py_ShellExecuteA((HWND)0, operation, filepath, + NULL, NULL, SW_SHOWNORMAL); Py_END_ALLOW_THREADS if (rc <= (HINSTANCE)32) { PyObject *errval = win32_error("startfile", filepath); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 24 18:50:00 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 24 Jan 2015 17:50:00 +0000 Subject: [Python-checkins] =?utf-8?q?devguide=3A_Issue_23257=3A_Update_Win?= =?utf-8?q?dows_build/setup_instructions=2E?= Message-ID: <20150124174949.75784.24173@psf.io> https://hg.python.org/devguide/rev/41d2d182c260 changeset: 727:41d2d182c260 user: Steve Dower date: Sat Jan 17 09:11:55 2015 -0800 summary: Issue 23257: Update Windows build/setup instructions. files: index.rst | 2 +- setup.rst | 52 +++++++++++++++++++++--------------------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -32,7 +32,7 @@ ./python -m test -j3 On :ref:`most ` Mac OS X systems, replace :file:`./python` - with :file:`./python.exe`. On Windows, use :file:`PCbuild\\python_d.exe` or + with :file:`./python.exe`. On Windows, use :file:`python.bat` or check the :ref:`Windows instructions `. With Python 2.7, replace ``test`` with ``test.regrtest``. 4. Make the :doc:`patch `. diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -226,12 +226,17 @@ included in the solution has more details, especially on what additional software is required to build which parts of Python. -**Python 3.3** and later use Microsoft Visual Studio 2010. You can -download Microsoft Visual C++ 2010 Express `from Microsoft's site +**Python 3.5** and later use Microsoft Visual Studio 2015. You can download +`Microsoft Visual Studio 2015 Ultimate Preview +`_, or you can +continue to use Visual Studio 2010 SP1 until Microsoft releases the final +version of Visual Studio 2015. + +Python 3.3 and 3.4 use Microsoft Visual Studio 2010. You can download +Microsoft Visual C++ 2010 Express `from Microsoft's site `_. To use it for more than 28 days, one must register through a -Windows Live account. - +Microsoft account (formerly known as a Windows Live or Hotmail account). You'll also need to install the Visual Studio `Service Pack 1 (SP1) `_. If you don't install this service pack, you may receive errors like the following @@ -244,22 +249,18 @@ Regardless of Visual Studio version, the ``PCbuild`` directory of a source checkout contains the build files for the Python version you are building. -The full version of Visual Studio is not necessary for common tasks with -32-bit builds; the gratis C++ Express versions linked above are sufficient. -Their limitations are given `here (2008) -`_ -and `here (2010) -`_. -Because the Python solution file uses Solution Folders, VS Express will warn -you about using solution folders every time you open the ``pcbuild.sln`` file. -You can safely dismiss the warning with no impact on your ability to build -Python. + +To build from the command line, execute :file:`PCBuild\\build.bat`. If you +have not previously done so, you can pass the ``-e`` option to download +external dependencies or invoke :file:`PCBuild\\get_externals.bat` directly. By +default, :file:`PCBuild\\build.bat` will produce a 32-bit release build. Pass +the ``-p x64`` option to produce a 64-bit build, and/or the ``-d`` option to +produce a debug build. To build from the Visual Studio GUI, open the ``pcbuild.sln`` solution file with Visual Studio. Choose the :menuselection:`Build Solution` option -under the :menuselection:`Build` or :menuselection:`Debug` menu -(depending on your version of Visual Studio). Be sure that "Debug" was -chosen as the active solution configuration (e.g. under +under the :menuselection:`Build` menu. Be sure that "Debug" was chosen +as the active solution configuration (e.g. under :menuselection:`Build --> Configuration Manager...`). When building you may see a number of build errors related to missing @@ -271,18 +272,18 @@ dependencies. Once built you might want to set Python as a startup project. Pressing F5 in -Visual Studio, or choosing Start Debugging from the Debug menu, will launch -the interpreter. +Visual Studio, or choosing :menuselection:`Start Debugging` from the +:menuselection:`Debug` menu, will launch the interpreter. .. _win-python.exe: If you want to launch the compiled interpreter from the command-line, the path varies according to the build. For a 32-bit build in debug mode, you -have to invoke ``PCBuild\python_d.exe``, for a 64-bit build in debug mode, -``PCBuild\amd64\python_d.exe``. If you are compiling in release mode (which -you shouldn't, in general), replace ``python_d.exe`` with ``python.exe``. -You can also invoke the most recently built interpreter using ``python.bat`` -in the root of the source tree. +have to invoke ``PCBuild\win32\python_d.exe``, for a 64-bit build in debug +mode, ``PCBuild\amd64\python_d.exe``. If you are compiling in release mode +(which you shouldn't, in general), replace ``python_d.exe`` with +``python.exe``. You can also invoke the most recently built interpreter using +``python.bat`` in the root of the source tree. .. _build_troubleshooting: @@ -362,8 +363,7 @@ Code for all built-in types. ``PC`` - Windows-specific code along with legacy build files for previously used - versions of MSVC. + Windows-specific code. ``PCbuild`` Build files for the version of MSVC currently used for the Windows -- Repository URL: https://hg.python.org/devguide From python-checkins at python.org Sat Jan 24 21:59:01 2015 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 24 Jan 2015 20:59:01 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fix_Issue23300?= =?utf-8?q?_=3A_httplib=2EHTTP_classe=27s_connect_method_should_use_=5Fget?= =?utf-8?q?=5Fhostport?= Message-ID: <20150124205826.55116.94943@psf.io> https://hg.python.org/cpython/rev/c2bba3c9424b changeset: 94268:c2bba3c9424b branch: 2.7 parent: 94265:94ec4d8cf104 user: Senthil Kumaran date: Sat Jan 24 12:58:10 2015 -0800 summary: Fix Issue23300 : httplib.HTTP classe's connect method should use _get_hostport instead of (non-existing) _set_hostport. (Fix the regression introduced in 568041fd8090 ) files: Lib/httplib.py | 4 +- Lib/test/test_httplib.py | 29 ++++++++++++++++++++++++++- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/Lib/httplib.py b/Lib/httplib.py --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -723,7 +723,7 @@ endpoint passed to set_tunnel. This is done by sending a HTTP CONNECT request to the proxy server when the connection is established. - This method must be called before the HTML connection has been + This method must be called before the HTTP connection has been established. The headers argument should be a mapping of extra HTTP headers @@ -1129,7 +1129,7 @@ "Accept arguments to set the host/port, since the superclass doesn't." if host is not None: - self._conn._set_hostport(host, port) + (self._conn.host, self._conn.port) = self._conn._get_hostport(host, port) self._conn.connect() def getfile(self): diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -461,7 +461,11 @@ self.assertEqual(httplib.responses[httplib.NOT_FOUND], "Not Found") -class SourceAddressTest(TestCase): +class TestServerMixin: + """A limited socket server mixin. + + This is used by test cases for testing http connection end points. + """ def setUp(self): self.serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.port = test_support.bind_port(self.serv) @@ -476,6 +480,7 @@ self.serv.close() self.serv = None +class SourceAddressTest(TestServerMixin, TestCase): def testHTTPConnectionSourceAddress(self): self.conn = httplib.HTTPConnection(HOST, self.port, source_address=('', self.source_port)) @@ -492,6 +497,24 @@ # for an ssl_wrapped connect() to actually return from. +class HTTPTest(TestServerMixin, TestCase): + def testHTTPConnection(self): + self.conn = httplib.HTTP(host=HOST, port=self.port, strict=None) + self.conn.connect() + self.assertEqual(self.conn._conn.host, HOST) + self.assertEqual(self.conn._conn.port, self.port) + + def testHTTPWithConnectHostPort(self): + testhost = 'unreachable.test.domain' + testport = '80' + self.conn = httplib.HTTP(host=testhost, port=testport) + self.conn.connect(host=HOST, port=self.port) + self.assertNotEqual(self.conn._conn.host, testhost) + self.assertNotEqual(self.conn._conn.port, testport) + self.assertEqual(self.conn._conn.host, HOST) + self.assertEqual(self.conn._conn.port, self.port) + + class TimeoutTest(TestCase): PORT = None @@ -537,6 +560,7 @@ self.assertEqual(httpConn.sock.gettimeout(), 30) httpConn.close() + class HTTPSTest(TestCase): def setUp(self): @@ -713,7 +737,8 @@ @test_support.reap_threads def test_main(verbose=None): test_support.run_unittest(HeaderTests, OfflineTest, BasicTest, TimeoutTest, - HTTPSTest, SourceAddressTest, TunnelTests) + HTTPTest, HTTPSTest, SourceAddressTest, + TunnelTests) if __name__ == '__main__': test_main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 25 04:26:29 2015 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 25 Jan 2015 03:26:29 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Increase_http?= =?utf-8?q?=2Eclient=2EHTTPConnection_test_coverage=2E?= Message-ID: <20150125032624.87117.14690@psf.io> https://hg.python.org/cpython/rev/fcab9c106f2f changeset: 94269:fcab9c106f2f branch: 3.4 parent: 94259:93888975606b user: Senthil Kumaran date: Sat Jan 24 19:24:59 2015 -0800 summary: Increase http.client.HTTPConnection test coverage. Added a new tunnel test to verify setting of _tunnel_host, _tunnel_port, _tunnel_headers attributes on HTTPConnection object. files: Lib/test/test_httplib.py | 67 +++++++++++++++++---------- 1 files changed, 42 insertions(+), 25 deletions(-) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -1041,8 +1041,7 @@ self.assertEqual(header, 42) class TunnelTests(TestCase): - - def test_connect(self): + def setUp(self): response_text = ( 'HTTP/1.0 200 OK\r\n\r\n' # Reply to CONNECT 'HTTP/1.1 200 OK\r\n' # Reply to HEAD @@ -1050,38 +1049,56 @@ ) def create_connection(address, timeout=None, source_address=None): - return FakeSocket(response_text, host=address[0], - port=address[1]) + return FakeSocket(response_text, host=address[0], port=address[1]) - conn = client.HTTPConnection('proxy.com') - conn._create_connection = create_connection + self.host = 'proxy.com' + self.conn = client.HTTPConnection(self.host) + self.conn._create_connection = create_connection + def tearDown(self): + self.conn.close() + + def test_set_tunnel_host_port_headers(self): + tunnel_host = 'destination.com' + tunnel_port = 8888 + tunnel_headers = {'User-Agent': 'Mozilla/5.0 (compatible, MSIE 11)'} + self.conn.set_tunnel(tunnel_host, port=tunnel_port, + headers=tunnel_headers) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, client.HTTP_PORT) + self.assertEqual(self.conn._tunnel_host, tunnel_host) + self.assertEqual(self.conn._tunnel_port, tunnel_port) + self.assertEqual(self.conn._tunnel_headers, tunnel_headers) + + def test_disallow_set_tunnel_after_connect(self): # Once connected, we shouldn't be able to tunnel anymore - conn.connect() - self.assertRaises(RuntimeError, conn.set_tunnel, + self.conn.connect() + self.assertRaises(RuntimeError, self.conn.set_tunnel, 'destination.com') - # But if we close the connection, we're good - conn.close() - conn.set_tunnel('destination.com') - conn.request('HEAD', '/', '') - - self.assertEqual(conn.sock.host, 'proxy.com') - self.assertEqual(conn.sock.port, 80) - self.assertIn(b'CONNECT destination.com', conn.sock.data) + def test_connect_with_tunnel(self): + self.conn.set_tunnel('destination.com') + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, client.HTTP_PORT) + self.assertIn(b'CONNECT destination.com', self.conn.sock.data) # issue22095 - self.assertNotIn(b'Host: destination.com:None', conn.sock.data) - self.assertIn(b'Host: destination.com', conn.sock.data) + self.assertNotIn(b'Host: destination.com:None', self.conn.sock.data) + self.assertIn(b'Host: destination.com', self.conn.sock.data) # This test should be removed when CONNECT gets the HTTP/1.1 blessing - self.assertNotIn(b'Host: proxy.com', conn.sock.data) + self.assertNotIn(b'Host: proxy.com', self.conn.sock.data) - conn.close() - conn.request('PUT', '/', '') - self.assertEqual(conn.sock.host, 'proxy.com') - self.assertEqual(conn.sock.port, 80) - self.assertTrue(b'CONNECT destination.com' in conn.sock.data) - self.assertTrue(b'Host: destination.com' in conn.sock.data) + def test_connect_put_request(self): + self.conn.set_tunnel('destination.com') + self.conn.request('PUT', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, client.HTTP_PORT) + self.assertIn(b'CONNECT destination.com', self.conn.sock.data) + self.assertIn(b'Host: destination.com', self.conn.sock.data) + + @support.reap_threads def test_main(verbose=None): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 25 04:26:28 2015 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 25 Jan 2015 03:26:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_merge_from_3=2E4?= Message-ID: <20150125032625.84255.52750@psf.io> https://hg.python.org/cpython/rev/a858cde113f2 changeset: 94270:a858cde113f2 parent: 94267:5bff604a864e parent: 94269:fcab9c106f2f user: Senthil Kumaran date: Sat Jan 24 19:26:18 2015 -0800 summary: merge from 3.4 Increase http.client.HTTPConnection test coverage. Added a new tunnel test to verify setting of _tunnel_host, _tunnel_port, _tunnel_headers attributes on HTTPConnection object. files: Lib/test/test_httplib.py | 67 +++++++++++++++++---------- 1 files changed, 42 insertions(+), 25 deletions(-) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -1253,8 +1253,7 @@ self.assertEqual(header, 42) class TunnelTests(TestCase): - - def test_connect(self): + def setUp(self): response_text = ( 'HTTP/1.0 200 OK\r\n\r\n' # Reply to CONNECT 'HTTP/1.1 200 OK\r\n' # Reply to HEAD @@ -1262,38 +1261,56 @@ ) def create_connection(address, timeout=None, source_address=None): - return FakeSocket(response_text, host=address[0], - port=address[1]) + return FakeSocket(response_text, host=address[0], port=address[1]) - conn = client.HTTPConnection('proxy.com') - conn._create_connection = create_connection + self.host = 'proxy.com' + self.conn = client.HTTPConnection(self.host) + self.conn._create_connection = create_connection + def tearDown(self): + self.conn.close() + + def test_set_tunnel_host_port_headers(self): + tunnel_host = 'destination.com' + tunnel_port = 8888 + tunnel_headers = {'User-Agent': 'Mozilla/5.0 (compatible, MSIE 11)'} + self.conn.set_tunnel(tunnel_host, port=tunnel_port, + headers=tunnel_headers) + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, client.HTTP_PORT) + self.assertEqual(self.conn._tunnel_host, tunnel_host) + self.assertEqual(self.conn._tunnel_port, tunnel_port) + self.assertEqual(self.conn._tunnel_headers, tunnel_headers) + + def test_disallow_set_tunnel_after_connect(self): # Once connected, we shouldn't be able to tunnel anymore - conn.connect() - self.assertRaises(RuntimeError, conn.set_tunnel, + self.conn.connect() + self.assertRaises(RuntimeError, self.conn.set_tunnel, 'destination.com') - # But if we close the connection, we're good - conn.close() - conn.set_tunnel('destination.com') - conn.request('HEAD', '/', '') - - self.assertEqual(conn.sock.host, 'proxy.com') - self.assertEqual(conn.sock.port, 80) - self.assertIn(b'CONNECT destination.com', conn.sock.data) + def test_connect_with_tunnel(self): + self.conn.set_tunnel('destination.com') + self.conn.request('HEAD', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, client.HTTP_PORT) + self.assertIn(b'CONNECT destination.com', self.conn.sock.data) # issue22095 - self.assertNotIn(b'Host: destination.com:None', conn.sock.data) - self.assertIn(b'Host: destination.com', conn.sock.data) + self.assertNotIn(b'Host: destination.com:None', self.conn.sock.data) + self.assertIn(b'Host: destination.com', self.conn.sock.data) # This test should be removed when CONNECT gets the HTTP/1.1 blessing - self.assertNotIn(b'Host: proxy.com', conn.sock.data) + self.assertNotIn(b'Host: proxy.com', self.conn.sock.data) - conn.close() - conn.request('PUT', '/', '') - self.assertEqual(conn.sock.host, 'proxy.com') - self.assertEqual(conn.sock.port, 80) - self.assertTrue(b'CONNECT destination.com' in conn.sock.data) - self.assertTrue(b'Host: destination.com' in conn.sock.data) + def test_connect_put_request(self): + self.conn.set_tunnel('destination.com') + self.conn.request('PUT', '/', '') + self.assertEqual(self.conn.sock.host, self.host) + self.assertEqual(self.conn.sock.port, client.HTTP_PORT) + self.assertIn(b'CONNECT destination.com', self.conn.sock.data) + self.assertIn(b'Host: destination.com', self.conn.sock.data) + + @support.reap_threads def test_main(verbose=None): -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Sun Jan 25 08:53:24 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 25 Jan 2015 08:53:24 +0100 Subject: [Python-checkins] Daily reference leaks (5bff604a864e): sum=3 Message-ID: results for 5bff604a864e on branch "default" -------------------------------------------- test_collections leaked [-2, 0, 2] references, sum=0 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogPz_hfe', '-x'] From python-checkins at python.org Sun Jan 25 21:49:56 2015 From: python-checkins at python.org (r.david.murray) Date: Sun, 25 Jan 2015 20:49:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzIzMjUxOiBSZWZs?= =?utf-8?q?ow_paragraph=2E?= Message-ID: <20150125204945.118210.54297@psf.io> https://hg.python.org/cpython/rev/5db28a3199b2 changeset: 94275:5db28a3199b2 branch: 2.7 user: R David Murray date: Sun Jan 25 15:48:47 2015 -0500 summary: #23251: Reflow paragraph. files: Doc/library/time.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -223,12 +223,12 @@ .. function:: sleep(secs) Suspend execution of the current thread for the given number of seconds. - The argument may be a - floating point number to indicate a more precise sleep time. The actual - suspension time may be less than that requested because any caught signal will - terminate the :func:`sleep` following execution of that signal's catching - routine. Also, the suspension time may be longer than requested by an arbitrary - amount because of the scheduling of other activity in the system. + The argument may be a floating point number to indicate a more precise sleep + time. The actual suspension time may be less than that requested because any + caught signal will terminate the :func:`sleep` following execution of that + signal's catching routine. Also, the suspension time may be longer than + requested by an arbitrary amount because of the scheduling of other activity + in the system. .. function:: strftime(format[, t]) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 25 21:49:56 2015 From: python-checkins at python.org (r.david.murray) Date: Sun, 25 Jan 2015 20:49:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzIzMjUxOiBub3Rl?= =?utf-8?q?_that_time=2Esleep_affects_the_current_thread_only=2E?= Message-ID: <20150125204945.87133.87051@psf.io> https://hg.python.org/cpython/rev/5e01c68cabbf changeset: 94274:5e01c68cabbf branch: 2.7 parent: 94268:c2bba3c9424b user: R David Murray date: Sun Jan 25 15:48:26 2015 -0500 summary: #23251: note that time.sleep affects the current thread only. This is parallel to the language used in the unix man page. files: Doc/library/time.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -222,7 +222,8 @@ .. function:: sleep(secs) - Suspend execution for the given number of seconds. The argument may be a + Suspend execution of the current thread for the given number of seconds. + The argument may be a floating point number to indicate a more precise sleep time. The actual suspension time may be less than that requested because any caught signal will terminate the :func:`sleep` following execution of that signal's catching -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 25 21:49:56 2015 From: python-checkins at python.org (r.david.murray) Date: Sun, 25 Jan 2015 20:49:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogIzIzMjE1OiByZWZs?= =?utf-8?q?ow_paragraph=2E?= Message-ID: <20150125204944.55126.30277@psf.io> https://hg.python.org/cpython/rev/3a9b1e5fe179 changeset: 94272:3a9b1e5fe179 branch: 3.4 user: R David Murray date: Sun Jan 25 15:46:22 2015 -0500 summary: #23215: reflow paragraph. files: Doc/library/time.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -343,12 +343,12 @@ .. function:: sleep(secs) Suspend execution of the calling thread for the given number of seconds. - The argument may be a - floating point number to indicate a more precise sleep time. The actual - suspension time may be less than that requested because any caught signal will - terminate the :func:`sleep` following execution of that signal's catching - routine. Also, the suspension time may be longer than requested by an arbitrary - amount because of the scheduling of other activity in the system. + The argument may be a floating point number to indicate a more precise sleep + time. The actual suspension time may be less than that requested because any + caught signal will terminate the :func:`sleep` following execution of that + signal's catching routine. Also, the suspension time may be longer than + requested by an arbitrary amount because of the scheduling of other activity + in the system. .. function:: strftime(format[, t]) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 25 21:49:56 2015 From: python-checkins at python.org (r.david.murray) Date: Sun, 25 Jan 2015 20:49:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogIzIzMjUxOiBOb3Rl?= =?utf-8?q?_that_time=2Esleep_affects_the_calling_thread_only=2E?= Message-ID: <20150125204944.87107.48459@psf.io> https://hg.python.org/cpython/rev/55ad65c4f9e2 changeset: 94271:55ad65c4f9e2 branch: 3.4 parent: 94269:fcab9c106f2f user: R David Murray date: Sun Jan 25 15:45:14 2015 -0500 summary: #23251: Note that time.sleep affects the calling thread only. This change parallels the language used in the unix man page. files: Doc/library/time.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -342,7 +342,8 @@ .. function:: sleep(secs) - Suspend execution for the given number of seconds. The argument may be a + Suspend execution of the calling thread for the given number of seconds. + The argument may be a floating point number to indicate a more precise sleep time. The actual suspension time may be less than that requested because any caught signal will terminate the :func:`sleep` following execution of that signal's catching -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 25 21:49:56 2015 From: python-checkins at python.org (r.david.murray) Date: Sun, 25 Jan 2015 20:49:56 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_=2323215=3A_note_that_time=2Esleep_affects_the_?= =?utf-8?q?current_thread_only=2E?= Message-ID: <20150125204945.75780.63020@psf.io> https://hg.python.org/cpython/rev/52a06812d5da changeset: 94273:52a06812d5da parent: 94270:a858cde113f2 parent: 94272:3a9b1e5fe179 user: R David Murray date: Sun Jan 25 15:47:06 2015 -0500 summary: Merge: #23215: note that time.sleep affects the current thread only. files: Doc/library/time.rst | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -342,12 +342,13 @@ .. function:: sleep(secs) - Suspend execution for the given number of seconds. The argument may be a - floating point number to indicate a more precise sleep time. The actual - suspension time may be less than that requested because any caught signal will - terminate the :func:`sleep` following execution of that signal's catching - routine. Also, the suspension time may be longer than requested by an arbitrary - amount because of the scheduling of other activity in the system. + Suspend execution of the calling thread for the given number of seconds. + The argument may be a floating point number to indicate a more precise sleep + time. The actual suspension time may be less than that requested because any + caught signal will terminate the :func:`sleep` following execution of that + signal's catching routine. Also, the suspension time may be longer than + requested by an arbitrary amount because of the scheduling of other activity + in the system. .. function:: strftime(format[, t]) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Jan 25 21:58:02 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 25 Jan 2015 20:58:02 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322286=3A_The_=22b?= =?utf-8?q?ackslashreplace=22_error_handlers_now_works_with?= Message-ID: <20150125205744.118224.25269@psf.io> https://hg.python.org/cpython/rev/dd8a03e98158 changeset: 94276:dd8a03e98158 parent: 94273:52a06812d5da user: Serhiy Storchaka date: Sun Jan 25 22:56:57 2015 +0200 summary: Issue #22286: The "backslashreplace" error handlers now works with decoding and translating. files: Doc/howto/unicode.rst | 7 +- Doc/library/codecs.rst | 14 +- Doc/library/functions.rst | 5 +- Doc/library/io.rst | 11 +- Doc/whatsnew/3.5.rst | 4 +- Lib/codecs.py | 9 +- Lib/test/test_codeccallbacks.py | 26 +- Lib/test/test_codecs.py | 56 +++++++ Misc/NEWS | 3 + Python/codecs.c | 152 ++++++++++++------- 10 files changed, 200 insertions(+), 87 deletions(-) diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -280,8 +280,9 @@ The *errors* argument specifies the response when the input string can't be converted according to the encoding's rules. Legal values for this argument are ``'strict'`` (raise a :exc:`UnicodeDecodeError` exception), ``'replace'`` (use -``U+FFFD``, ``REPLACEMENT CHARACTER``), or ``'ignore'`` (just leave the -character out of the Unicode result). +``U+FFFD``, ``REPLACEMENT CHARACTER``), ``'ignore'`` (just leave the +character out of the Unicode result), or ``'backslashreplace'`` (inserts a +``\xNN`` escape sequence). The following examples show the differences:: >>> b'\x80abc'.decode("utf-8", "strict") #doctest: +NORMALIZE_WHITESPACE @@ -291,6 +292,8 @@ invalid start byte >>> b'\x80abc'.decode("utf-8", "replace") '\ufffdabc' + >>> b'\x80abc'.decode("utf-8", "backslashreplace") + '\\x80abc' >>> b'\x80abc'.decode("utf-8", "ignore") 'abc' diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -314,8 +314,8 @@ | | reference (only for encoding). Implemented | | | in :func:`xmlcharrefreplace_errors`. | +-------------------------+-----------------------------------------------+ -| ``'backslashreplace'`` | Replace with backslashed escape sequences | -| | (only for encoding). Implemented in | +| ``'backslashreplace'`` | Replace with backslashed escape sequences. | +| | Implemented in | | | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ | ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | @@ -350,6 +350,10 @@ .. versionadded:: 3.5 The ``'namereplace'`` error handler. +.. versionchanged:: 3.5 + The ``'backslashreplace'`` error handlers now works with decoding and + translating. + The set of allowed values can be extended by registering a new named error handler: @@ -417,9 +421,9 @@ .. function:: backslashreplace_errors(exception) - Implements the ``'backslashreplace'`` error handling (for encoding with - :term:`text encodings ` only): the - unencodable character is replaced by a backslashed escape sequence. + Implements the ``'backslashreplace'`` error handling (for + :term:`text encodings ` only): malformed data is + replaced by a backslashed escape sequence. .. function:: namereplace_errors(exception) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -973,9 +973,8 @@ Characters not supported by the encoding are replaced with the appropriate XML character reference ``&#nnn;``. - * ``'backslashreplace'`` (also only supported when writing) - replaces unsupported characters with Python's backslashed escape - sequences. + * ``'backslashreplace'`` replaces malformed data by Python's backslashed + escape sequences. * ``'namereplace'`` (also only supported when writing) replaces unsupported characters with ``\N{...}`` escape sequences. diff --git a/Doc/library/io.rst b/Doc/library/io.rst --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -825,11 +825,12 @@ exception if there is an encoding error (the default of ``None`` has the same effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding errors can lead to data loss.) ``'replace'`` causes a replacement marker - (such as ``'?'``) to be inserted where there is malformed data. When - writing, ``'xmlcharrefreplace'`` (replace with the appropriate XML character - reference), ``'backslashreplace'`` (replace with backslashed escape - sequences) or ``'namereplace'`` (replace with ``\N{...}`` escape sequences) - can be used. Any other error handling name that has been registered with + (such as ``'?'``) to be inserted where there is malformed data. + ``'backslashreplace'`` causes malformed data to be replaced by a + backslashed escape sequence. When writing, ``'xmlcharrefreplace'`` + (replace with the appropriate XML character reference) or ``'namereplace'`` + (replace with ``\N{...}`` escape sequences) can be used. Any other error + handling name that has been registered with :func:`codecs.register_error` is also valid. .. index:: diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -118,7 +118,9 @@ Some smaller changes made to the core Python language are: -* None yet. +* Added the ``'namereplace'`` error handlers. The ``'backslashreplace'`` + error handlers now works with decoding and translating. + (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) diff --git a/Lib/codecs.py b/Lib/codecs.py --- a/Lib/codecs.py +++ b/Lib/codecs.py @@ -127,7 +127,8 @@ 'surrogateescape' - replace with private code points U+DCnn. 'xmlcharrefreplace' - Replace with the appropriate XML character reference (only for encoding). - 'backslashreplace' - Replace with backslashed escape sequences + 'backslashreplace' - Replace with backslashed escape sequences. + 'namereplace' - Replace with \\N{...} escape sequences (only for encoding). The set of allowed values can be extended via register_error. @@ -359,7 +360,8 @@ 'xmlcharrefreplace' - Replace with the appropriate XML character reference. 'backslashreplace' - Replace with backslashed escape - sequences (only for encoding). + sequences. + 'namereplace' - Replace with \\N{...} escape sequences. The set of allowed parameter values can be extended via register_error. @@ -429,7 +431,8 @@ 'strict' - raise a ValueError (or a subclass) 'ignore' - ignore the character and continue with the next - 'replace'- replace with a suitable replacement character; + 'replace'- replace with a suitable replacement character + 'backslashreplace' - Replace with backslashed escape sequences; The set of allowed parameter values can be extended via register_error. diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py --- a/Lib/test/test_codeccallbacks.py +++ b/Lib/test/test_codeccallbacks.py @@ -246,6 +246,11 @@ "\u0000\ufffd" ) + self.assertEqual( + b"\x00\x00\x00\x00\x00".decode("unicode-internal", "backslashreplace"), + "\u0000\\x00" + ) + codecs.register_error("test.hui", handler_unicodeinternal) self.assertEqual( @@ -565,17 +570,6 @@ codecs.backslashreplace_errors, UnicodeError("ouch") ) - # "backslashreplace" can only be used for encoding - self.assertRaises( - TypeError, - codecs.backslashreplace_errors, - UnicodeDecodeError("ascii", bytearray(b"\xff"), 0, 1, "ouch") - ) - self.assertRaises( - TypeError, - codecs.backslashreplace_errors, - UnicodeTranslateError("\u3042", 0, 1, "ouch") - ) # Use the correct exception self.assertEqual( codecs.backslashreplace_errors( @@ -701,6 +695,16 @@ UnicodeEncodeError("ascii", "\udfff", 0, 1, "ouch")), ("\\udfff", 1) ) + self.assertEqual( + codecs.backslashreplace_errors( + UnicodeDecodeError("ascii", bytearray(b"\xff"), 0, 1, "ouch")), + ("\\xff", 1) + ) + self.assertEqual( + codecs.backslashreplace_errors( + UnicodeTranslateError("\u3042", 0, 1, "ouch")), + ("\\u3042", 1) + ) def test_badhandlerresults(self): results = ( 42, "foo", (1,2,3), ("foo", 1, 3), ("foo", None), ("foo",), ("foo", 1, 3), ("foo", None), ("foo",) ) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -378,6 +378,10 @@ before + after) self.assertEqual(test_sequence.decode(self.encoding, "replace"), before + self.ill_formed_sequence_replace + after) + backslashreplace = ''.join('\\x%02x' % b + for b in self.ill_formed_sequence) + self.assertEqual(test_sequence.decode(self.encoding, "backslashreplace"), + before + backslashreplace + after) class UTF32Test(ReadTest, unittest.TestCase): encoding = "utf-32" @@ -1300,14 +1304,19 @@ "unicode_internal") if sys.byteorder == "little": invalid = b"\x00\x00\x11\x00" + invalid_backslashreplace = r"\x00\x00\x11\x00" else: invalid = b"\x00\x11\x00\x00" + invalid_backslashreplace = r"\x00\x11\x00\x00" with support.check_warnings(): self.assertRaises(UnicodeDecodeError, invalid.decode, "unicode_internal") with support.check_warnings(): self.assertEqual(invalid.decode("unicode_internal", "replace"), '\ufffd') + with support.check_warnings(): + self.assertEqual(invalid.decode("unicode_internal", "backslashreplace"), + invalid_backslashreplace) @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t') def test_decode_error_attributes(self): @@ -2043,6 +2052,16 @@ ) self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "backslashreplace", "ab"), + ("ab\\x02", 3) + ) + + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "backslashreplace", "ab\ufffe"), + ("ab\\x02", 3) + ) + + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", "ab"), ("ab", 3) ) @@ -2119,6 +2138,25 @@ ) self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "backslashreplace", + {0: 'a', 1: 'b'}), + ("ab\\x02", 3) + ) + + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "backslashreplace", + {0: 'a', 1: 'b', 2: None}), + ("ab\\x02", 3) + ) + + # Issue #14850 + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "backslashreplace", + {0: 'a', 1: 'b', 2: '\ufffe'}), + ("ab\\x02", 3) + ) + + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", {0: 'a', 1: 'b'}), ("ab", 3) @@ -2195,6 +2233,18 @@ ) self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "backslashreplace", + {0: a, 1: b}), + ("ab\\x02", 3) + ) + + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "backslashreplace", + {0: a, 1: b, 2: 0xFFFE}), + ("ab\\x02", 3) + ) + + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", {0: a, 1: b}), ("ab", 3) @@ -2253,9 +2303,13 @@ self.assertRaises(UnicodeDecodeError, codecs.unicode_escape_decode, br"\U00110000") self.assertEqual(codecs.unicode_escape_decode(r"\U00110000", "replace"), ("\ufffd", 10)) + self.assertEqual(codecs.unicode_escape_decode(r"\U00110000", "backslashreplace"), + (r"\x5c\x55\x30\x30\x31\x31\x30\x30\x30\x30", 10)) self.assertRaises(UnicodeDecodeError, codecs.raw_unicode_escape_decode, br"\U00110000") self.assertEqual(codecs.raw_unicode_escape_decode(r"\U00110000", "replace"), ("\ufffd", 10)) + self.assertEqual(codecs.raw_unicode_escape_decode(r"\U00110000", "backslashreplace"), + (r"\x5c\x55\x30\x30\x31\x31\x30\x30\x30\x30", 10)) class UnicodeEscapeTest(unittest.TestCase): @@ -2894,11 +2948,13 @@ (b'[\xff]', 'strict', None), (b'[\xff]', 'ignore', '[]'), (b'[\xff]', 'replace', '[\ufffd]'), + (b'[\xff]', 'backslashreplace', '[\\xff]'), (b'[\xff]', 'surrogateescape', '[\udcff]'), (b'[\xff]', 'surrogatepass', None), (b'\x81\x00abc', 'strict', None), (b'\x81\x00abc', 'ignore', '\x00abc'), (b'\x81\x00abc', 'replace', '\ufffd\x00abc'), + (b'\x81\x00abc', 'backslashreplace', '\\xff\x00abc'), )) def test_cp1252(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #22286: The "backslashreplace" error handlers now works with + decoding and translating. + - Issue #23253: Delay-load ShellExecute[AW] in os.startfile for reduced startup overhead on Windows. diff --git a/Python/codecs.c b/Python/codecs.c --- a/Python/codecs.c +++ b/Python/codecs.c @@ -864,74 +864,112 @@ PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc) { + PyObject *object; + Py_ssize_t i; + Py_ssize_t start; + Py_ssize_t end; + PyObject *res; + unsigned char *outp; + int ressize; + Py_UCS4 c; + + if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) { + unsigned char *p; + if (PyUnicodeDecodeError_GetStart(exc, &start)) + return NULL; + if (PyUnicodeDecodeError_GetEnd(exc, &end)) + return NULL; + if (!(object = PyUnicodeDecodeError_GetObject(exc))) + return NULL; + if (!(p = (unsigned char*)PyBytes_AsString(object))) { + Py_DECREF(object); + return NULL; + } + res = PyUnicode_New(4 * (end - start), 127); + if (res == NULL) { + Py_DECREF(object); + return NULL; + } + outp = PyUnicode_1BYTE_DATA(res); + for (i = start; i < end; i++, outp += 4) { + unsigned char c = p[i]; + outp[0] = '\\'; + outp[1] = 'x'; + outp[2] = Py_hexdigits[(c>>4)&0xf]; + outp[3] = Py_hexdigits[c&0xf]; + } + + assert(_PyUnicode_CheckConsistency(res, 1)); + Py_DECREF(object); + return Py_BuildValue("(Nn)", res, end); + } if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) { - PyObject *restuple; - PyObject *object; - Py_ssize_t i; - Py_ssize_t start; - Py_ssize_t end; - PyObject *res; - unsigned char *outp; - Py_ssize_t ressize; - Py_UCS4 c; if (PyUnicodeEncodeError_GetStart(exc, &start)) return NULL; if (PyUnicodeEncodeError_GetEnd(exc, &end)) return NULL; if (!(object = PyUnicodeEncodeError_GetObject(exc))) return NULL; - if (end - start > PY_SSIZE_T_MAX / (1+1+8)) - end = start + PY_SSIZE_T_MAX / (1+1+8); - for (i = start, ressize = 0; i < end; ++i) { - /* object is guaranteed to be "ready" */ - c = PyUnicode_READ_CHAR(object, i); - if (c >= 0x10000) { - ressize += 1+1+8; - } - else if (c >= 0x100) { - ressize += 1+1+4; - } - else - ressize += 1+1+2; - } - res = PyUnicode_New(ressize, 127); - if (res == NULL) { - Py_DECREF(object); + } + else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) { + if (PyUnicodeTranslateError_GetStart(exc, &start)) return NULL; - } - for (i = start, outp = PyUnicode_1BYTE_DATA(res); - i < end; ++i) { - c = PyUnicode_READ_CHAR(object, i); - *outp++ = '\\'; - if (c >= 0x00010000) { - *outp++ = 'U'; - *outp++ = Py_hexdigits[(c>>28)&0xf]; - *outp++ = Py_hexdigits[(c>>24)&0xf]; - *outp++ = Py_hexdigits[(c>>20)&0xf]; - *outp++ = Py_hexdigits[(c>>16)&0xf]; - *outp++ = Py_hexdigits[(c>>12)&0xf]; - *outp++ = Py_hexdigits[(c>>8)&0xf]; - } - else if (c >= 0x100) { - *outp++ = 'u'; - *outp++ = Py_hexdigits[(c>>12)&0xf]; - *outp++ = Py_hexdigits[(c>>8)&0xf]; - } - else - *outp++ = 'x'; - *outp++ = Py_hexdigits[(c>>4)&0xf]; - *outp++ = Py_hexdigits[c&0xf]; - } - - assert(_PyUnicode_CheckConsistency(res, 1)); - restuple = Py_BuildValue("(Nn)", res, end); - Py_DECREF(object); - return restuple; + if (PyUnicodeTranslateError_GetEnd(exc, &end)) + return NULL; + if (!(object = PyUnicodeTranslateError_GetObject(exc))) + return NULL; } else { wrong_exception_type(exc); return NULL; } + + if (end - start > PY_SSIZE_T_MAX / (1+1+8)) + end = start + PY_SSIZE_T_MAX / (1+1+8); + for (i = start, ressize = 0; i < end; ++i) { + /* object is guaranteed to be "ready" */ + c = PyUnicode_READ_CHAR(object, i); + if (c >= 0x10000) { + ressize += 1+1+8; + } + else if (c >= 0x100) { + ressize += 1+1+4; + } + else + ressize += 1+1+2; + } + res = PyUnicode_New(ressize, 127); + if (res == NULL) { + Py_DECREF(object); + return NULL; + } + outp = PyUnicode_1BYTE_DATA(res); + for (i = start; i < end; ++i) { + c = PyUnicode_READ_CHAR(object, i); + *outp++ = '\\'; + if (c >= 0x00010000) { + *outp++ = 'U'; + *outp++ = Py_hexdigits[(c>>28)&0xf]; + *outp++ = Py_hexdigits[(c>>24)&0xf]; + *outp++ = Py_hexdigits[(c>>20)&0xf]; + *outp++ = Py_hexdigits[(c>>16)&0xf]; + *outp++ = Py_hexdigits[(c>>12)&0xf]; + *outp++ = Py_hexdigits[(c>>8)&0xf]; + } + else if (c >= 0x100) { + *outp++ = 'u'; + *outp++ = Py_hexdigits[(c>>12)&0xf]; + *outp++ = Py_hexdigits[(c>>8)&0xf]; + } + else + *outp++ = 'x'; + *outp++ = Py_hexdigits[(c>>4)&0xf]; + *outp++ = Py_hexdigits[c&0xf]; + } + + assert(_PyUnicode_CheckConsistency(res, 1)); + Py_DECREF(object); + return Py_BuildValue("(Nn)", res, end); } static _PyUnicode_Name_CAPI *ucnhash_CAPI = NULL; @@ -1444,8 +1482,8 @@ backslashreplace_errors, METH_O, PyDoc_STR("Implements the 'backslashreplace' error handling, " - "which replaces an unencodable character with a " - "backslashed escape sequence.") + "which replaces malformed data with a backslashed " + "escape sequence.") } }, { -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 00:27:43 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 25 Jan 2015 23:27:43 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323321=3A_Fixed_a_crash_in_str=2Edecode=28=29_wh?= =?utf-8?q?en_error_handler_returned?= Message-ID: <20150125232742.87109.57319@psf.io> https://hg.python.org/cpython/rev/1cd68b3c46aa changeset: 94278:1cd68b3c46aa parent: 94276:dd8a03e98158 parent: 94277:2de90090e486 user: Serhiy Storchaka date: Mon Jan 26 01:24:31 2015 +0200 summary: Issue #23321: Fixed a crash in str.decode() when error handler returned replacment string longer than mailformed input data. files: Misc/NEWS | 3 +++ Objects/unicodeobject.c | 8 ++++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #23321: Fixed a crash in str.decode() when error handler returned + replacment string longer than mailformed input data. + - Issue #22286: The "backslashreplace" error handlers now works with decoding and translating. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -4155,9 +4155,13 @@ if (PyUnicode_READY(repunicode) < 0) goto onError; replen = PyUnicode_GET_LENGTH(repunicode); - writer->min_length += replen; - if (replen > 1) + if (replen > 1) { + writer->min_length += replen - 1; writer->overallocate = 1; + if (_PyUnicodeWriter_Prepare(writer, writer->min_length, + PyUnicode_MAX_CHAR_VALUE(repunicode)) == -1) + goto onError; + } if (_PyUnicodeWriter_WriteStr(writer, repunicode) == -1) goto onError; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 00:27:42 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 25 Jan 2015 23:27:42 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzIx?= =?utf-8?q?=3A_Fixed_a_crash_in_str=2Edecode=28=29_when_error_handler_retu?= =?utf-8?q?rned?= Message-ID: <20150125232742.75796.63204@psf.io> https://hg.python.org/cpython/rev/2de90090e486 changeset: 94277:2de90090e486 branch: 3.4 parent: 94272:3a9b1e5fe179 user: Serhiy Storchaka date: Mon Jan 26 01:22:54 2015 +0200 summary: Issue #23321: Fixed a crash in str.decode() when error handler returned replacment string longer than mailformed input data. files: Misc/NEWS | 3 +++ Objects/unicodeobject.c | 8 ++++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -11,6 +11,9 @@ Core and Builtins ----------------- +- Issue #23321: Fixed a crash in str.decode() when error handler returned + replacment string longer than mailformed input data. + - Issue #23048: Fix jumping out of an infinite while loop in the pdb. - Issue #20335: bytes constructor now raises TypeError when encoding or errors diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -4190,9 +4190,13 @@ if (PyUnicode_READY(repunicode) < 0) goto onError; replen = PyUnicode_GET_LENGTH(repunicode); - writer->min_length += replen; - if (replen > 1) + if (replen > 1) { + writer->min_length += replen - 1; writer->overallocate = 1; + if (_PyUnicodeWriter_Prepare(writer, writer->min_length, + PyUnicode_MAX_CHAR_VALUE(repunicode)) == -1) + goto onError; + } if (_PyUnicodeWriter_WriteStr(writer, repunicode) == -1) goto onError; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 01:13:15 2015 From: python-checkins at python.org (raymond.hettinger) Date: Mon, 26 Jan 2015 00:13:15 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323119=3A__Simplif?= =?utf-8?q?y_setobject_by_inlining_the_special_case_for_unicode?= Message-ID: <20150126001257.75776.75177@psf.io> https://hg.python.org/cpython/rev/20f54cdf351d changeset: 94279:20f54cdf351d user: Raymond Hettinger date: Sun Jan 25 16:12:49 2015 -0800 summary: Issue #23119: Simplify setobject by inlining the special case for unicode equality testing. files: Include/setobject.h | 3 +- Lib/test/test_sys.py | 2 +- Objects/setobject.c | 81 ++++--------------------------- 3 files changed, 13 insertions(+), 73 deletions(-) diff --git a/Include/setobject.h b/Include/setobject.h --- a/Include/setobject.h +++ b/Include/setobject.h @@ -35,7 +35,7 @@ */ -typedef struct _setobject { +typedef struct { PyObject_HEAD Py_ssize_t fill; /* Number active and dummy entries*/ @@ -53,7 +53,6 @@ * runtime null-tests. */ setentry *table; - setentry *(*lookup)(struct _setobject *so, PyObject *key, Py_hash_t hash); Py_hash_t hash; /* Only used by frozenset objects */ setentry smalltable[PySet_MINSIZE]; diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -994,7 +994,7 @@ # frozenset PySet_MINSIZE = 8 samples = [[], range(10), range(50)] - s = size('3n2P' + PySet_MINSIZE*'nP' + '2nP') + s = size('3nP' + PySet_MINSIZE*'nP' + '2nP') for sample in samples: minused = len(sample) if minused == 0: tmp = 1 diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -69,6 +69,10 @@ PyObject *startkey = entry->key; if (startkey == key) return entry; + if (PyUnicode_CheckExact(startkey) + && PyUnicode_CheckExact(key) + && unicode_eq(startkey, key)) + return entry; Py_INCREF(startkey); cmp = PyObject_RichCompareBool(startkey, key, Py_EQ); Py_DECREF(startkey); @@ -90,6 +94,10 @@ PyObject *startkey = entry->key; if (startkey == key) return entry; + if (PyUnicode_CheckExact(startkey) + && PyUnicode_CheckExact(key) + && unicode_eq(startkey, key)) + return entry; Py_INCREF(startkey); cmp = PyObject_RichCompareBool(startkey, key, Py_EQ); Py_DECREF(startkey); @@ -116,68 +124,6 @@ } /* - * Hacked up version of set_lookkey which can assume keys are always unicode; - * This means we can always use unicode_eq directly and not have to check to - * see if the comparison altered the table. - */ -static setentry * -set_lookkey_unicode(PySetObject *so, PyObject *key, Py_hash_t hash) -{ - setentry *table = so->table; - setentry *freeslot = NULL; - setentry *entry; - size_t perturb = hash; - size_t mask = so->mask; - size_t i = (size_t)hash; - size_t j; - - /* Make sure this function doesn't have to handle non-unicode keys, - including subclasses of str; e.g., one reason to subclass - strings is to override __eq__, and for speed we don't cater to - that here. */ - if (!PyUnicode_CheckExact(key)) { /* unlikely */ - so->lookup = set_lookkey; - return set_lookkey(so, key, hash); - } - - entry = &table[i & mask]; - if (entry->key == NULL) - return entry; - - while (1) { - if (entry->hash == hash - && (entry->key == key - || (entry->key != dummy /* unlikely */ - && unicode_eq(entry->key, key)))) /* likely */ - return entry; - if (entry->key == dummy && freeslot == NULL) - freeslot = entry; - - for (j = 1 ; j <= LINEAR_PROBES ; j++) { - entry = &table[(i + j) & mask]; - if (entry->key == NULL) - goto found_null; - if (entry->hash == hash - && (entry->key == key - || (entry->key != dummy /* unlikely */ - && unicode_eq(entry->key, key)))) /* likely */ - return entry; - if (entry->key == dummy && freeslot == NULL) - freeslot = entry; - } - - perturb >>= PERTURB_SHIFT; - i = i * 5 + 1 + perturb; - - entry = &table[i & mask]; - if (entry->key == NULL) - goto found_null; - } - found_null: - return freeslot == NULL ? entry : freeslot; -} - -/* Internal routine used by set_table_resize() to insert an item which is known to be absent from the set. This routine also assumes that the set contains no deleted entries. Besides the performance benefit, @@ -225,8 +171,7 @@ { setentry *entry; - assert(so->lookup != NULL); - entry = so->lookup(so, key, hash); + entry = set_lookkey(so, key, hash); if (entry == NULL) return -1; if (entry->key == NULL) { @@ -385,7 +330,7 @@ setentry *entry; PyObject *old_key; - entry = (so->lookup)(so, oldentry->key, oldentry->hash); + entry = set_lookkey(so, oldentry->key, oldentry->hash); if (entry == NULL) return -1; if (entry->key == NULL || entry->key == dummy) @@ -631,7 +576,7 @@ PyObject *key; setentry *lu_entry; - lu_entry = (so->lookup)(so, entry->key, entry->hash); + lu_entry = set_lookkey(so, entry->key, entry->hash); if (lu_entry == NULL) return -1; key = lu_entry->key; @@ -994,7 +939,6 @@ so->used = 0; so->mask = PySet_MINSIZE - 1; so->table = so->smalltable; - so->lookup = set_lookkey_unicode; so->hash = -1; so->finger = 0; so->weakreflist = NULL; @@ -1095,7 +1039,6 @@ { Py_ssize_t t; setentry *u; - setentry *(*f)(PySetObject *so, PyObject *key, Py_ssize_t hash); setentry tab[PySet_MINSIZE]; Py_hash_t h; @@ -1111,8 +1054,6 @@ a->table = a->smalltable; b->table = u; - f = a->lookup; a->lookup = b->lookup; b->lookup = f; - if (a->table == a->smalltable || b->table == b->smalltable) { memcpy(tab, a->smalltable, sizeof(tab)); memcpy(a->smalltable, b->smalltable, sizeof(tab)); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 01:44:20 2015 From: python-checkins at python.org (raymond.hettinger) Date: Mon, 26 Jan 2015 00:44:20 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Set_the_hash_values_of_dum?= =?utf-8?q?my_entries_to_-1=2E__Improves_quality_of_entry-=3Ehash_=3D=3D?= Message-ID: <20150126004401.55106.50251@psf.io> https://hg.python.org/cpython/rev/a2da8cf05f48 changeset: 94281:a2da8cf05f48 user: Raymond Hettinger date: Sun Jan 25 16:38:52 2015 -0800 summary: Set the hash values of dummy entries to -1. Improves quality of entry->hash == hash tests. files: Objects/setobject.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -337,6 +337,7 @@ return DISCARD_NOTFOUND; old_key = entry->key; entry->key = dummy; + entry->hash = -1; so->used--; Py_DECREF(old_key); return DISCARD_FOUND; @@ -621,6 +622,7 @@ } key = entry->key; entry->key = dummy; + entry->hash = -1; so->used--; so->finger = i + 1; /* next place to start */ return key; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 01:44:19 2015 From: python-checkins at python.org (raymond.hettinger) Date: Mon, 26 Jan 2015 00:44:19 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Update_out-of-date_comment?= =?utf-8?q?s=2E?= Message-ID: <20150126004401.55128.86857@psf.io> https://hg.python.org/cpython/rev/b132091ad95b changeset: 94280:b132091ad95b user: Raymond Hettinger date: Sun Jan 25 16:27:40 2015 -0800 summary: Update out-of-date comments. files: Objects/setobject.c | 8 +++----- 1 files changed, 3 insertions(+), 5 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -23,7 +23,7 @@ All arithmetic on hash should ignore overflow. - Unlike the dictionary implementation, the lookkey functions can return + Unlike the dictionary implementation, the lookkey function can return NULL if the rich comparison returns an error. */ @@ -1028,10 +1028,8 @@ t=set(a); a.clear(); a.update(b); b.clear(); b.update(t); del t The function always succeeds and it leaves both objects in a stable state. - Useful for creating temporary frozensets from sets for membership testing - in __contains__(), discard(), and remove(). Also useful for operations - that update in-place (by allowing an intermediate result to be swapped - into one of the original inputs). + Useful for operations that update in-place (by allowing an intermediate + result to be swapped into one of the original inputs). */ static void -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 05:35:21 2015 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 26 Jan 2015 04:35:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_handle_headers?= =?utf-8?q?_with_no_key_=28closes_=2319996=29?= Message-ID: <20150126043448.55106.24090@psf.io> https://hg.python.org/cpython/rev/25ecf3d0ea03 changeset: 94282:25ecf3d0ea03 branch: 3.4 parent: 94277:2de90090e486 user: Benjamin Peterson date: Sun Jan 25 23:30:30 2015 -0500 summary: handle headers with no key (closes #19996) Patch by Cory Benfield. files: Lib/email/feedparser.py | 11 ++++++++++- Lib/test/test_email/test_email.py | 6 ++++++ Lib/test/test_httplib.py | 10 ++++++++++ Misc/NEWS | 3 +++ 4 files changed, 29 insertions(+), 1 deletions(-) diff --git a/Lib/email/feedparser.py b/Lib/email/feedparser.py --- a/Lib/email/feedparser.py +++ b/Lib/email/feedparser.py @@ -33,7 +33,7 @@ NLCRE_crack = re.compile('(\r\n|\r|\n)') # RFC 2822 $3.6.8 Optional fields. ftext is %d33-57 / %d59-126, Any character # except controls, SP, and ":". -headerRE = re.compile(r'^(From |[\041-\071\073-\176]{1,}:|[\t ])') +headerRE = re.compile(r'^(From |[\041-\071\073-\176]*:|[\t ])') EMPTYSTRING = '' NL = '\n' @@ -511,6 +511,15 @@ # There will always be a colon, because if there wasn't the part of # the parser that calls us would have started parsing the body. i = line.find(':') + + # If the colon is on the start of the line the header is clearly + # malformed, but we might be able to salvage the rest of the + # message. Track the error but keep going. + if i == 0: + defect = errors.InvalidHeaderDefect("Missing header name.") + self._cur.defects.append(defect) + continue + assert i>0, "_parse_headers fed line with no : and no leading WS" lastheader = line[:i] lastvalue = [line] diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -3389,6 +3389,12 @@ feedparser.feed(chunk) return feedparser.close() + def test_empty_header_name_handled(self): + # Issue 19996 + msg = self.parse("First: val\n: bad\nSecond: val") + self.assertEqual(msg['First'], 'val') + self.assertEqual(msg['Second'], 'val') + def test_newlines(self): m = self.parse(['a:\nb:\rc:\r\nd:\n']) self.assertEqual(m.keys(), ['a', 'b', 'c', 'd']) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -167,6 +167,16 @@ conn.request('GET', '/foo') self.assertTrue(sock.data.startswith(expected)) + def test_malformed_headers_coped_with(self): + # Issue 19996 + body = "HTTP/1.1 200 OK\r\nFirst: val\r\n: nval\r\nSecond: val\r\n\r\n" + sock = FakeSocket(body) + resp = client.HTTPResponse(sock) + resp.begin() + + self.assertEqual(resp.getheader('First'), 'val') + self.assertEqual(resp.getheader('Second'), 'val') + class BasicTest(TestCase): def test_status_lines(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -47,6 +47,9 @@ Library ------- +- Issue #19996: :class:`email.feedparser.FeedParser` now handles (malformed) + headers with no key rather than amusing the body has started. + - Issue #23248: Update ssl error codes from latest OpenSSL git master. - Issue #23098: 64-bit dev_t is now supported in the os module. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 05:35:21 2015 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 26 Jan 2015 04:35:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_simply_ignore_?= =?utf-8?q?headers_with_no_name_=28=2319996=29?= Message-ID: <20150126043449.75778.32880@psf.io> https://hg.python.org/cpython/rev/9a4882b12218 changeset: 94284:9a4882b12218 branch: 2.7 parent: 94275:5db28a3199b2 user: Benjamin Peterson date: Sun Jan 25 23:34:42 2015 -0500 summary: simply ignore headers with no name (#19996) Patch by Cory Benfield. files: Lib/httplib.py | 5 +++++ Lib/rfc822.py | 7 ++++++- Lib/test/test_httplib.py | 10 ++++++++++ Lib/test/test_rfc822.py | 6 ++++++ Misc/NEWS | 3 +++ 5 files changed, 30 insertions(+), 1 deletions(-) diff --git a/Lib/httplib.py b/Lib/httplib.py --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -313,6 +313,11 @@ hlist.append(line) self.addheader(headerseen, line[len(headerseen)+1:].strip()) continue + elif headerseen is not None: + # An empty header name. These aren't allowed in HTTP, but it's + # probably a benign mistake. Don't add the header, just keep + # going. + continue else: # It's not a header line; throw it back and stop here. if not self.dict: diff --git a/Lib/rfc822.py b/Lib/rfc822.py --- a/Lib/rfc822.py +++ b/Lib/rfc822.py @@ -179,6 +179,11 @@ lst.append(line) self.dict[headerseen] = line[len(headerseen)+1:].strip() continue + elif headerseen is not None: + # An empty header name. These aren't allowed in HTTP, but it's + # probably a benign mistake. Don't add the header, just keep + # going. + continue else: # It's not a header line; throw it back and stop here. if not self.dict: @@ -202,7 +207,7 @@ data in RFC 2822-like formats with special header formats. """ i = line.find(':') - if i > 0: + if i > -1: return line[:i].lower() return None diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -164,6 +164,16 @@ conn.request('GET', '/foo') self.assertTrue(sock.data.startswith(expected)) + def test_malformed_headers_coped_with(self): + # Issue 19996 + body = "HTTP/1.1 200 OK\r\nFirst: val\r\n: nval\r\nSecond: val\r\n\r\n" + sock = FakeSocket(body) + resp = httplib.HTTPResponse(sock) + resp.begin() + + self.assertEqual(resp.getheader('First'), 'val') + self.assertEqual(resp.getheader('Second'), 'val') + class BasicTest(TestCase): def test_status_lines(self): diff --git a/Lib/test/test_rfc822.py b/Lib/test/test_rfc822.py --- a/Lib/test/test_rfc822.py +++ b/Lib/test/test_rfc822.py @@ -248,6 +248,12 @@ eq(rfc822.quote('foo\\wacky"name'), 'foo\\\\wacky\\"name') eq(rfc822.unquote('"foo\\\\wacky\\"name"'), 'foo\\wacky"name') + def test_invalid_headers(self): + eq = self.assertEqual + msg = self.create_message("First: val\n: otherval\nSecond: val2\n") + eq(msg.getheader('First'), 'val') + eq(msg.getheader('Second'), 'val2') + def test_main(): test_support.run_unittest(MessageTestCase) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,9 @@ Library ------- +- Issue #19996: Make :mod:`httplib` ignore headers with no name rather than + assuming the body has started. + - Issue #20188: Support Application-Layer Protocol Negotiation (ALPN) in the ssl module. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 05:35:21 2015 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 26 Jan 2015 04:35:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMTk5OTYp?= Message-ID: <20150126043449.87129.83066@psf.io> https://hg.python.org/cpython/rev/29923a9987be changeset: 94283:29923a9987be parent: 94281:a2da8cf05f48 parent: 94282:25ecf3d0ea03 user: Benjamin Peterson date: Sun Jan 25 23:31:58 2015 -0500 summary: merge 3.4 (#19996) files: Lib/email/feedparser.py | 11 ++++++++++- Lib/test/test_email/test_email.py | 6 ++++++ Lib/test/test_httplib.py | 10 ++++++++++ Misc/NEWS | 3 +++ 4 files changed, 29 insertions(+), 1 deletions(-) diff --git a/Lib/email/feedparser.py b/Lib/email/feedparser.py --- a/Lib/email/feedparser.py +++ b/Lib/email/feedparser.py @@ -33,7 +33,7 @@ NLCRE_crack = re.compile('(\r\n|\r|\n)') # RFC 2822 $3.6.8 Optional fields. ftext is %d33-57 / %d59-126, Any character # except controls, SP, and ":". -headerRE = re.compile(r'^(From |[\041-\071\073-\176]{1,}:|[\t ])') +headerRE = re.compile(r'^(From |[\041-\071\073-\176]*:|[\t ])') EMPTYSTRING = '' NL = '\n' @@ -511,6 +511,15 @@ # There will always be a colon, because if there wasn't the part of # the parser that calls us would have started parsing the body. i = line.find(':') + + # If the colon is on the start of the line the header is clearly + # malformed, but we might be able to salvage the rest of the + # message. Track the error but keep going. + if i == 0: + defect = errors.InvalidHeaderDefect("Missing header name.") + self._cur.defects.append(defect) + continue + assert i>0, "_parse_headers fed line with no : and no leading WS" lastheader = line[:i] lastvalue = [line] diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -3393,6 +3393,12 @@ feedparser.feed(chunk) return feedparser.close() + def test_empty_header_name_handled(self): + # Issue 19996 + msg = self.parse("First: val\n: bad\nSecond: val") + self.assertEqual(msg['First'], 'val') + self.assertEqual(msg['Second'], 'val') + def test_newlines(self): m = self.parse(['a:\nb:\rc:\r\nd:\n']) self.assertEqual(m.keys(), ['a', 'b', 'c', 'd']) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -190,6 +190,16 @@ conn.request('GET', '/foo') self.assertTrue(sock.data.startswith(expected)) + def test_malformed_headers_coped_with(self): + # Issue 19996 + body = "HTTP/1.1 200 OK\r\nFirst: val\r\n: nval\r\nSecond: val\r\n\r\n" + sock = FakeSocket(body) + resp = client.HTTPResponse(sock) + resp.begin() + + self.assertEqual(resp.getheader('First'), 'val') + self.assertEqual(resp.getheader('Second'), 'val') + class BasicTest(TestCase): def test_status_lines(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -215,6 +215,9 @@ Library ------- +- Issue #19996: :class:`email.feedparser.FeedParser` now handles (malformed) + headers with no key rather than amusing the body has started. + - Issue #20188: Support Application-Layer Protocol Negotiation (ALPN) in the ssl module. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 08:28:55 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 07:28:55 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Don=27t_use_deprecated_ass?= =?utf-8?q?ertEquals=2E?= Message-ID: <20150126072854.118204.26205@psf.io> https://hg.python.org/cpython/rev/4f80d36ef185 changeset: 94285:4f80d36ef185 parent: 94283:29923a9987be user: Serhiy Storchaka date: Mon Jan 26 09:28:11 2015 +0200 summary: Don't use deprecated assertEquals. files: Lib/test/test_bytes.py | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -470,7 +470,7 @@ self.assertFalse(b is orig) b = b'%s / 100 = %d%%' a = b % (b'seventy-nine', 79) - self.assertEquals(a, b'seventy-nine / 100 = 79%') + self.assertEqual(a, b'seventy-nine / 100 = 79%') def test_imod(self): b = b'hello, %b!' @@ -481,7 +481,7 @@ self.assertFalse(b is orig) b = b'%s / 100 = %d%%' b %= (b'seventy-nine', 79) - self.assertEquals(b, b'seventy-nine / 100 = 79%') + self.assertEqual(b, b'seventy-nine / 100 = 79%') def test_replace(self): b = self.type2test(b'mississippi') @@ -1021,7 +1021,7 @@ self.assertFalse(b is orig) b = bytearray(b'%s / 100 = %d%%') a = b % (b'seventy-nine', 79) - self.assertEquals(a, bytearray(b'seventy-nine / 100 = 79%')) + self.assertEqual(a, bytearray(b'seventy-nine / 100 = 79%')) def test_imod(self): b = bytearray(b'hello, %b!') @@ -1032,7 +1032,7 @@ self.assertFalse(b is orig) b = bytearray(b'%s / 100 = %d%%') b %= (b'seventy-nine', 79) - self.assertEquals(b, bytearray(b'seventy-nine / 100 = 79%')) + self.assertEqual(b, bytearray(b'seventy-nine / 100 = 79%')) def test_iconcat(self): b = bytearray(b"abc") -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:05:16 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:05:16 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDA4?= =?utf-8?q?=3A_The_default_=5F=5Fne=5F=5F=28=29_now_returns_NotImplemented?= =?utf-8?b?IGlmIF9fZXFfXygp?= Message-ID: <20150126080511.118212.87967@psf.io> https://hg.python.org/cpython/rev/e516badfd3b2 changeset: 94286:e516badfd3b2 branch: 3.4 parent: 94282:25ecf3d0ea03 user: Serhiy Storchaka date: Mon Jan 26 09:57:07 2015 +0200 summary: Issue #21408: The default __ne__() now returns NotImplemented if __eq__() returned NotImplemented. Removed incorrect implementations of __ne__(). files: Lib/_collections_abc.py | 6 -- Lib/doctest.py | 9 --- Lib/lib2to3/pytree.py | 10 ---- Lib/numbers.py | 5 -- Lib/pathlib.py | 3 - Lib/test/test_binop.py | 4 - Lib/test/test_compare.py | 65 +++++++++++++++++++++++++++- Lib/unittest/case.py | 3 - Lib/unittest/suite.py | 3 - Misc/NEWS | 7 +++ Objects/typeobject.c | 9 +++- 11 files changed, 77 insertions(+), 47 deletions(-) diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py --- a/Lib/_collections_abc.py +++ b/Lib/_collections_abc.py @@ -224,9 +224,6 @@ return NotImplemented return len(self) == len(other) and self.__le__(other) - def __ne__(self, other): - return not (self == other) - @classmethod def _from_iterable(cls, it): '''Construct an instance of the class from any iterable input. @@ -451,9 +448,6 @@ return NotImplemented return dict(self.items()) == dict(other.items()) - def __ne__(self, other): - return not (self == other) - Mapping.register(mappingproxy) diff --git a/Lib/doctest.py b/Lib/doctest.py --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -481,9 +481,6 @@ self.options == other.options and \ self.exc_msg == other.exc_msg - def __ne__(self, other): - return not self == other - def __hash__(self): return hash((self.source, self.want, self.lineno, self.indent, self.exc_msg)) @@ -547,9 +544,6 @@ self.filename == other.filename and \ self.lineno == other.lineno - def __ne__(self, other): - return not self == other - def __hash__(self): return hash((self.docstring, self.name, self.filename, self.lineno)) @@ -2289,9 +2283,6 @@ self._dt_tearDown == other._dt_tearDown and \ self._dt_checker == other._dt_checker - def __ne__(self, other): - return not self == other - def __hash__(self): return hash((self._dt_optionflags, self._dt_setUp, self._dt_tearDown, self._dt_checker)) diff --git a/Lib/lib2to3/pytree.py b/Lib/lib2to3/pytree.py --- a/Lib/lib2to3/pytree.py +++ b/Lib/lib2to3/pytree.py @@ -64,16 +64,6 @@ __hash__ = None # For Py3 compatibility. - def __ne__(self, other): - """ - Compare two nodes for inequality. - - This calls the method _eq(). - """ - if self.__class__ is not other.__class__: - return NotImplemented - return not self._eq(other) - def _eq(self, other): """ Compare two nodes for equality. diff --git a/Lib/numbers.py b/Lib/numbers.py --- a/Lib/numbers.py +++ b/Lib/numbers.py @@ -141,11 +141,6 @@ """self == other""" raise NotImplementedError - def __ne__(self, other): - """self != other""" - # The default __ne__ doesn't negate __eq__ until 3.0. - return not (self == other) - Complex.register(complex) diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -665,9 +665,6 @@ return NotImplemented return self._cparts == other._cparts and self._flavour is other._flavour - def __ne__(self, other): - return not self == other - def __hash__(self): try: return self._hash diff --git a/Lib/test/test_binop.py b/Lib/test/test_binop.py --- a/Lib/test/test_binop.py +++ b/Lib/test/test_binop.py @@ -194,10 +194,6 @@ return float(self) == other return NotImplemented - def __ne__(self, other): - """Compare two Rats for inequality.""" - return not self == other - class RatTestCase(unittest.TestCase): """Unit tests for Rat class and its support utilities.""" diff --git a/Lib/test/test_compare.py b/Lib/test/test_compare.py --- a/Lib/test/test_compare.py +++ b/Lib/test/test_compare.py @@ -48,8 +48,69 @@ def test_ne_defaults_to_not_eq(self): a = Cmp(1) b = Cmp(1) - self.assertTrue(a == b) - self.assertFalse(a != b) + c = Cmp(2) + self.assertIs(a == b, True) + self.assertIs(a != b, False) + self.assertIs(a != c, True) + + def test_ne_high_priority(self): + """object.__ne__() should allow reflected __ne__() to be tried""" + calls = [] + class Left: + # Inherits object.__ne__() + def __eq__(*args): + calls.append('Left.__eq__') + return NotImplemented + class Right: + def __eq__(*args): + calls.append('Right.__eq__') + return NotImplemented + def __ne__(*args): + calls.append('Right.__ne__') + return NotImplemented + Left() != Right() + self.assertSequenceEqual(calls, ['Left.__eq__', 'Right.__ne__']) + + def test_ne_low_priority(self): + """object.__ne__() should not invoke reflected __eq__()""" + calls = [] + class Base: + # Inherits object.__ne__() + def __eq__(*args): + calls.append('Base.__eq__') + return NotImplemented + class Derived(Base): # Subclassing forces higher priority + def __eq__(*args): + calls.append('Derived.__eq__') + return NotImplemented + def __ne__(*args): + calls.append('Derived.__ne__') + return NotImplemented + Base() != Derived() + self.assertSequenceEqual(calls, ['Derived.__ne__', 'Base.__eq__']) + + def test_other_delegation(self): + """No default delegation between operations except __ne__()""" + ops = ( + ('__eq__', lambda a, b: a == b), + ('__lt__', lambda a, b: a < b), + ('__le__', lambda a, b: a <= b), + ('__gt__', lambda a, b: a > b), + ('__ge__', lambda a, b: a >= b), + ) + for name, func in ops: + with self.subTest(name): + def unexpected(*args): + self.fail('Unexpected operator method called') + class C: + __ne__ = unexpected + for other, _ in ops: + if other != name: + setattr(C, other, unexpected) + if name == '__eq__': + self.assertIs(func(C(), object()), False) + else: + self.assertRaises(TypeError, func, C(), object()) def test_issue_1393(self): x = lambda: None diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -1342,9 +1342,6 @@ self._testFunc == other._testFunc and \ self._description == other._description - def __ne__(self, other): - return not self == other - def __hash__(self): return hash((type(self), self._setUpFunc, self._tearDownFunc, self._testFunc, self._description)) diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py --- a/Lib/unittest/suite.py +++ b/Lib/unittest/suite.py @@ -31,9 +31,6 @@ return NotImplemented return list(self) == list(other) - def __ne__(self, other): - return not self == other - def __iter__(self): return iter(self._tests) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -11,6 +11,9 @@ Core and Builtins ----------------- +- Issue #21408: The default __ne__() now returns NotImplemented if __eq__() + returned NotImplemented. + - Issue #23321: Fixed a crash in str.decode() when error handler returned replacment string longer than mailformed input data. @@ -47,6 +50,10 @@ Library ------- +- Issue #21408: Removed incorrect implementations of __ne__() which didn't + returned NotImplemented if __eq__() returned NotImplemented. The default + __ne__() now works correctly. + - Issue #19996: :class:`email.feedparser.FeedParser` now handles (malformed) headers with no key rather than amusing the body has started. diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3348,9 +3348,14 @@ break; case Py_NE: - /* By default, != returns the opposite of ==, + /* By default, __ne__() delegates to __eq__() and inverts the result, unless the latter returns NotImplemented. */ - res = PyObject_RichCompare(self, other, Py_EQ); + if (self->ob_type->tp_richcompare == NULL) { + res = Py_NotImplemented; + Py_INCREF(res); + break; + } + res = (*self->ob_type->tp_richcompare)(self, other, Py_EQ); if (res != NULL && res != Py_NotImplemented) { int ok = PyObject_IsTrue(res); Py_DECREF(res); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:05:22 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:05:22 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogSXNzdWUgIzIxNDA4OiBUaGUgZGVmYXVsdCBfX25lX18oKSBub3cgcmV0?= =?utf-8?q?urns_NotImplemented_if_=5F=5Feq=5F=5F=28=29?= Message-ID: <20150126080511.75774.55758@psf.io> https://hg.python.org/cpython/rev/7e9880052401 changeset: 94287:7e9880052401 parent: 94285:4f80d36ef185 parent: 94286:e516badfd3b2 user: Serhiy Storchaka date: Mon Jan 26 10:04:15 2015 +0200 summary: Issue #21408: The default __ne__() now returns NotImplemented if __eq__() returned NotImplemented. Removed incorrect implementations of __ne__(). files: Lib/_collections_abc.py | 6 -- Lib/doctest.py | 9 --- Lib/lib2to3/pytree.py | 10 ---- Lib/numbers.py | 5 -- Lib/pathlib.py | 3 - Lib/test/test_binop.py | 4 - Lib/test/test_compare.py | 65 +++++++++++++++++++++++++++- Lib/unittest/case.py | 3 - Lib/unittest/suite.py | 3 - Misc/NEWS | 7 +++ Objects/typeobject.c | 9 +++- 11 files changed, 77 insertions(+), 47 deletions(-) diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py --- a/Lib/_collections_abc.py +++ b/Lib/_collections_abc.py @@ -224,9 +224,6 @@ return NotImplemented return len(self) == len(other) and self.__le__(other) - def __ne__(self, other): - return not (self == other) - @classmethod def _from_iterable(cls, it): '''Construct an instance of the class from any iterable input. @@ -451,9 +448,6 @@ return NotImplemented return dict(self.items()) == dict(other.items()) - def __ne__(self, other): - return not (self == other) - Mapping.register(mappingproxy) diff --git a/Lib/doctest.py b/Lib/doctest.py --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -481,9 +481,6 @@ self.options == other.options and \ self.exc_msg == other.exc_msg - def __ne__(self, other): - return not self == other - def __hash__(self): return hash((self.source, self.want, self.lineno, self.indent, self.exc_msg)) @@ -548,9 +545,6 @@ self.filename == other.filename and \ self.lineno == other.lineno - def __ne__(self, other): - return not self == other - def __hash__(self): return hash((self.docstring, self.name, self.filename, self.lineno)) @@ -2291,9 +2285,6 @@ self._dt_tearDown == other._dt_tearDown and \ self._dt_checker == other._dt_checker - def __ne__(self, other): - return not self == other - def __hash__(self): return hash((self._dt_optionflags, self._dt_setUp, self._dt_tearDown, self._dt_checker)) diff --git a/Lib/lib2to3/pytree.py b/Lib/lib2to3/pytree.py --- a/Lib/lib2to3/pytree.py +++ b/Lib/lib2to3/pytree.py @@ -64,16 +64,6 @@ __hash__ = None # For Py3 compatibility. - def __ne__(self, other): - """ - Compare two nodes for inequality. - - This calls the method _eq(). - """ - if self.__class__ is not other.__class__: - return NotImplemented - return not self._eq(other) - def _eq(self, other): """ Compare two nodes for equality. diff --git a/Lib/numbers.py b/Lib/numbers.py --- a/Lib/numbers.py +++ b/Lib/numbers.py @@ -141,11 +141,6 @@ """self == other""" raise NotImplementedError - def __ne__(self, other): - """self != other""" - # The default __ne__ doesn't negate __eq__ until 3.0. - return not (self == other) - Complex.register(complex) diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -710,9 +710,6 @@ return NotImplemented return self._cparts == other._cparts and self._flavour is other._flavour - def __ne__(self, other): - return not self == other - def __hash__(self): try: return self._hash diff --git a/Lib/test/test_binop.py b/Lib/test/test_binop.py --- a/Lib/test/test_binop.py +++ b/Lib/test/test_binop.py @@ -194,10 +194,6 @@ return float(self) == other return NotImplemented - def __ne__(self, other): - """Compare two Rats for inequality.""" - return not self == other - class RatTestCase(unittest.TestCase): """Unit tests for Rat class and its support utilities.""" diff --git a/Lib/test/test_compare.py b/Lib/test/test_compare.py --- a/Lib/test/test_compare.py +++ b/Lib/test/test_compare.py @@ -48,8 +48,69 @@ def test_ne_defaults_to_not_eq(self): a = Cmp(1) b = Cmp(1) - self.assertTrue(a == b) - self.assertFalse(a != b) + c = Cmp(2) + self.assertIs(a == b, True) + self.assertIs(a != b, False) + self.assertIs(a != c, True) + + def test_ne_high_priority(self): + """object.__ne__() should allow reflected __ne__() to be tried""" + calls = [] + class Left: + # Inherits object.__ne__() + def __eq__(*args): + calls.append('Left.__eq__') + return NotImplemented + class Right: + def __eq__(*args): + calls.append('Right.__eq__') + return NotImplemented + def __ne__(*args): + calls.append('Right.__ne__') + return NotImplemented + Left() != Right() + self.assertSequenceEqual(calls, ['Left.__eq__', 'Right.__ne__']) + + def test_ne_low_priority(self): + """object.__ne__() should not invoke reflected __eq__()""" + calls = [] + class Base: + # Inherits object.__ne__() + def __eq__(*args): + calls.append('Base.__eq__') + return NotImplemented + class Derived(Base): # Subclassing forces higher priority + def __eq__(*args): + calls.append('Derived.__eq__') + return NotImplemented + def __ne__(*args): + calls.append('Derived.__ne__') + return NotImplemented + Base() != Derived() + self.assertSequenceEqual(calls, ['Derived.__ne__', 'Base.__eq__']) + + def test_other_delegation(self): + """No default delegation between operations except __ne__()""" + ops = ( + ('__eq__', lambda a, b: a == b), + ('__lt__', lambda a, b: a < b), + ('__le__', lambda a, b: a <= b), + ('__gt__', lambda a, b: a > b), + ('__ge__', lambda a, b: a >= b), + ) + for name, func in ops: + with self.subTest(name): + def unexpected(*args): + self.fail('Unexpected operator method called') + class C: + __ne__ = unexpected + for other, _ in ops: + if other != name: + setattr(C, other, unexpected) + if name == '__eq__': + self.assertIs(func(C(), object()), False) + else: + self.assertRaises(TypeError, func, C(), object()) def test_issue_1393(self): x = lambda: None diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -1342,9 +1342,6 @@ self._testFunc == other._testFunc and \ self._description == other._description - def __ne__(self, other): - return not self == other - def __hash__(self): return hash((type(self), self._setUpFunc, self._tearDownFunc, self._testFunc, self._description)) diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py --- a/Lib/unittest/suite.py +++ b/Lib/unittest/suite.py @@ -31,9 +31,6 @@ return NotImplemented return list(self) == list(other) - def __ne__(self, other): - return not self == other - def __iter__(self): return iter(self._tests) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #21408: The default __ne__() now returns NotImplemented if __eq__() + returned NotImplemented. + - Issue #23321: Fixed a crash in str.decode() when error handler returned replacment string longer than mailformed input data. @@ -215,6 +218,10 @@ Library ------- +- Issue #21408: Removed incorrect implementations of __ne__() which didn't + returned NotImplemented if __eq__() returned NotImplemented. The default + __ne__() now works correctly. + - Issue #19996: :class:`email.feedparser.FeedParser` now handles (malformed) headers with no key rather than amusing the body has started. diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3385,9 +3385,14 @@ break; case Py_NE: - /* By default, != returns the opposite of ==, + /* By default, __ne__() delegates to __eq__() and inverts the result, unless the latter returns NotImplemented. */ - res = PyObject_RichCompare(self, other, Py_EQ); + if (self->ob_type->tp_richcompare == NULL) { + res = Py_NotImplemented; + Py_INCREF(res); + break; + } + res = (*self->ob_type->tp_richcompare)(self, other, Py_EQ); if (res != NULL && res != Py_NotImplemented) { int ok = PyObject_IsTrue(res); Py_DECREF(res); -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Mon Jan 26 09:11:07 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 26 Jan 2015 09:11:07 +0100 Subject: [Python-checkins] Daily reference leaks (a2da8cf05f48): sum=3 Message-ID: results for a2da8cf05f48 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogZDmTnw', '-x'] From python-checkins at python.org Mon Jan 26 09:12:21 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:12:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323268=3A_Fixed_bugs_in_the_comparison_of_ipaddr?= =?utf-8?q?ess_classes=2E?= Message-ID: <20150126081217.75796.55706@psf.io> https://hg.python.org/cpython/rev/051f2a234d8a changeset: 94289:051f2a234d8a parent: 94287:7e9880052401 parent: 94288:9a7d965ab80f user: Serhiy Storchaka date: Mon Jan 26 10:11:39 2015 +0200 summary: Issue #23268: Fixed bugs in the comparison of ipaddress classes. files: Lib/ipaddress.py | 47 +++---------------------- Lib/test/test_ipaddress.py | 47 +++++++++++++++++++++++-- Misc/NEWS | 2 + 3 files changed, 51 insertions(+), 45 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -382,40 +382,7 @@ return NotImplemented -class _TotalOrderingMixin: - # Helper that derives the other comparison operations from - # __lt__ and __eq__ - # We avoid functools.total_ordering because it doesn't handle - # NotImplemented correctly yet (http://bugs.python.org/issue10042) - def __eq__(self, other): - raise NotImplementedError - def __ne__(self, other): - equal = self.__eq__(other) - if equal is NotImplemented: - return NotImplemented - return not equal - def __lt__(self, other): - raise NotImplementedError - def __le__(self, other): - less = self.__lt__(other) - if less is NotImplemented or not less: - return self.__eq__(other) - return less - def __gt__(self, other): - less = self.__lt__(other) - if less is NotImplemented: - return NotImplemented - equal = self.__eq__(other) - if equal is NotImplemented: - return NotImplemented - return not (less or equal) - def __ge__(self, other): - less = self.__lt__(other) - if less is NotImplemented: - return NotImplemented - return not less - -class _IPAddressBase(_TotalOrderingMixin): +class _IPAddressBase: """The mother class.""" @@ -567,6 +534,7 @@ return self.__class__, (str(self),) + at functools.total_ordering class _BaseAddress(_IPAddressBase): """A generic IP object. @@ -586,12 +554,11 @@ return NotImplemented def __lt__(self, other): + if not isinstance(other, _BaseAddress): + return NotImplemented if self._version != other._version: raise TypeError('%s and %s are not of the same version' % ( self, other)) - if not isinstance(other, _BaseAddress): - raise TypeError('%s and %s are not of the same type' % ( - self, other)) if self._ip != other._ip: return self._ip < other._ip return False @@ -624,6 +591,7 @@ return self.__class__, (self._ip,) + at functools.total_ordering class _BaseNetwork(_IPAddressBase): """A generic IP network object. @@ -673,12 +641,11 @@ return self._address_class(broadcast + n) def __lt__(self, other): + if not isinstance(other, _BaseNetwork): + return NotImplemented if self._version != other._version: raise TypeError('%s and %s are not of the same version' % ( self, other)) - if not isinstance(other, _BaseNetwork): - raise TypeError('%s and %s are not of the same type' % ( - self, other)) if self.network_address != other.network_address: return self.network_address < other.network_address if self.netmask != other.netmask: diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -7,6 +7,7 @@ import unittest import re import contextlib +import functools import operator import pickle import ipaddress @@ -552,6 +553,20 @@ self.assertFactoryError(ipaddress.ip_network, "network") + at functools.total_ordering +class LargestObject: + def __eq__(self, other): + return isinstance(other, LargestObject) + def __lt__(self, other): + return False + + at functools.total_ordering +class SmallestObject: + def __eq__(self, other): + return isinstance(other, SmallestObject) + def __gt__(self, other): + return False + class ComparisonTests(unittest.TestCase): v4addr = ipaddress.IPv4Address(1) @@ -605,6 +620,28 @@ self.assertRaises(TypeError, lambda: lhs <= rhs) self.assertRaises(TypeError, lambda: lhs >= rhs) + def test_foreign_type_ordering(self): + other = object() + smallest = SmallestObject() + largest = LargestObject() + for obj in self.objects: + with self.assertRaises(TypeError): + obj < other + with self.assertRaises(TypeError): + obj > other + with self.assertRaises(TypeError): + obj <= other + with self.assertRaises(TypeError): + obj >= other + self.assertTrue(obj < largest) + self.assertFalse(obj > largest) + self.assertTrue(obj <= largest) + self.assertFalse(obj >= largest) + self.assertFalse(obj < smallest) + self.assertTrue(obj > smallest) + self.assertFalse(obj <= smallest) + self.assertTrue(obj >= smallest) + def test_mixed_type_key(self): # with get_mixed_type_key, you can sort addresses and network. v4_ordered = [self.v4addr, self.v4net, self.v4intf] @@ -625,7 +662,7 @@ v4addr = ipaddress.ip_address('1.1.1.1') v4net = ipaddress.ip_network('1.1.1.1') v6addr = ipaddress.ip_address('::1') - v6net = ipaddress.ip_address('::1') + v6net = ipaddress.ip_network('::1') self.assertRaises(TypeError, v4addr.__lt__, v6addr) self.assertRaises(TypeError, v4addr.__gt__, v6addr) @@ -1383,10 +1420,10 @@ unsorted = [ip4, ip1, ip3, ip2] unsorted.sort() self.assertEqual(sorted, unsorted) - self.assertRaises(TypeError, ip1.__lt__, - ipaddress.ip_address('10.10.10.0')) - self.assertRaises(TypeError, ip2.__lt__, - ipaddress.ip_address('10.10.10.0')) + self.assertIs(ip1.__lt__(ipaddress.ip_address('10.10.10.0')), + NotImplemented) + self.assertIs(ip2.__lt__(ipaddress.ip_address('10.10.10.0')), + NotImplemented) # <=, >= self.assertTrue(ipaddress.ip_network('1.1.1.1') <= diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -218,6 +218,8 @@ Library ------- +- Issue #23268: Fixed bugs in the comparison of ipaddress classes. + - Issue #21408: Removed incorrect implementations of __ne__() which didn't returned NotImplemented if __eq__() returned NotImplemented. The default __ne__() now works correctly. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:12:22 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:12:22 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjY4?= =?utf-8?q?=3A_Fixed_bugs_in_the_comparison_of_ipaddress_classes=2E?= Message-ID: <20150126081217.55100.35436@psf.io> https://hg.python.org/cpython/rev/9a7d965ab80f changeset: 94288:9a7d965ab80f branch: 3.4 parent: 94286:e516badfd3b2 user: Serhiy Storchaka date: Mon Jan 26 10:11:16 2015 +0200 summary: Issue #23268: Fixed bugs in the comparison of ipaddress classes. files: Lib/ipaddress.py | 47 +++---------------------- Lib/test/test_ipaddress.py | 47 +++++++++++++++++++++++-- Misc/NEWS | 2 + 3 files changed, 51 insertions(+), 45 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -388,40 +388,7 @@ return NotImplemented -class _TotalOrderingMixin: - # Helper that derives the other comparison operations from - # __lt__ and __eq__ - # We avoid functools.total_ordering because it doesn't handle - # NotImplemented correctly yet (http://bugs.python.org/issue10042) - def __eq__(self, other): - raise NotImplementedError - def __ne__(self, other): - equal = self.__eq__(other) - if equal is NotImplemented: - return NotImplemented - return not equal - def __lt__(self, other): - raise NotImplementedError - def __le__(self, other): - less = self.__lt__(other) - if less is NotImplemented or not less: - return self.__eq__(other) - return less - def __gt__(self, other): - less = self.__lt__(other) - if less is NotImplemented: - return NotImplemented - equal = self.__eq__(other) - if equal is NotImplemented: - return NotImplemented - return not (less or equal) - def __ge__(self, other): - less = self.__lt__(other) - if less is NotImplemented: - return NotImplemented - return not less - -class _IPAddressBase(_TotalOrderingMixin): +class _IPAddressBase: """The mother class.""" @@ -554,6 +521,7 @@ self._report_invalid_netmask(ip_str) + at functools.total_ordering class _BaseAddress(_IPAddressBase): """A generic IP object. @@ -578,12 +546,11 @@ return NotImplemented def __lt__(self, other): + if not isinstance(other, _BaseAddress): + return NotImplemented if self._version != other._version: raise TypeError('%s and %s are not of the same version' % ( self, other)) - if not isinstance(other, _BaseAddress): - raise TypeError('%s and %s are not of the same type' % ( - self, other)) if self._ip != other._ip: return self._ip < other._ip return False @@ -613,6 +580,7 @@ return (self._version, self) + at functools.total_ordering class _BaseNetwork(_IPAddressBase): """A generic IP network object. @@ -662,12 +630,11 @@ return self._address_class(broadcast + n) def __lt__(self, other): + if not isinstance(other, _BaseNetwork): + return NotImplemented if self._version != other._version: raise TypeError('%s and %s are not of the same version' % ( self, other)) - if not isinstance(other, _BaseNetwork): - raise TypeError('%s and %s are not of the same type' % ( - self, other)) if self.network_address != other.network_address: return self.network_address < other.network_address if self.netmask != other.netmask: diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -7,6 +7,7 @@ import unittest import re import contextlib +import functools import operator import ipaddress @@ -528,6 +529,20 @@ self.assertFactoryError(ipaddress.ip_network, "network") + at functools.total_ordering +class LargestObject: + def __eq__(self, other): + return isinstance(other, LargestObject) + def __lt__(self, other): + return False + + at functools.total_ordering +class SmallestObject: + def __eq__(self, other): + return isinstance(other, SmallestObject) + def __gt__(self, other): + return False + class ComparisonTests(unittest.TestCase): v4addr = ipaddress.IPv4Address(1) @@ -581,6 +596,28 @@ self.assertRaises(TypeError, lambda: lhs <= rhs) self.assertRaises(TypeError, lambda: lhs >= rhs) + def test_foreign_type_ordering(self): + other = object() + smallest = SmallestObject() + largest = LargestObject() + for obj in self.objects: + with self.assertRaises(TypeError): + obj < other + with self.assertRaises(TypeError): + obj > other + with self.assertRaises(TypeError): + obj <= other + with self.assertRaises(TypeError): + obj >= other + self.assertTrue(obj < largest) + self.assertFalse(obj > largest) + self.assertTrue(obj <= largest) + self.assertFalse(obj >= largest) + self.assertFalse(obj < smallest) + self.assertTrue(obj > smallest) + self.assertFalse(obj <= smallest) + self.assertTrue(obj >= smallest) + def test_mixed_type_key(self): # with get_mixed_type_key, you can sort addresses and network. v4_ordered = [self.v4addr, self.v4net, self.v4intf] @@ -601,7 +638,7 @@ v4addr = ipaddress.ip_address('1.1.1.1') v4net = ipaddress.ip_network('1.1.1.1') v6addr = ipaddress.ip_address('::1') - v6net = ipaddress.ip_address('::1') + v6net = ipaddress.ip_network('::1') self.assertRaises(TypeError, v4addr.__lt__, v6addr) self.assertRaises(TypeError, v4addr.__gt__, v6addr) @@ -1248,10 +1285,10 @@ unsorted = [ip4, ip1, ip3, ip2] unsorted.sort() self.assertEqual(sorted, unsorted) - self.assertRaises(TypeError, ip1.__lt__, - ipaddress.ip_address('10.10.10.0')) - self.assertRaises(TypeError, ip2.__lt__, - ipaddress.ip_address('10.10.10.0')) + self.assertIs(ip1.__lt__(ipaddress.ip_address('10.10.10.0')), + NotImplemented) + self.assertIs(ip2.__lt__(ipaddress.ip_address('10.10.10.0')), + NotImplemented) # <=, >= self.assertTrue(ipaddress.ip_network('1.1.1.1') <= diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -50,6 +50,8 @@ Library ------- +- Issue #23268: Fixed bugs in the comparison of ipaddress classes. + - Issue #21408: Removed incorrect implementations of __ne__() which didn't returned NotImplemented if __eq__() returned NotImplemented. The default __ne__() now works correctly. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:15:30 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:15:30 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Add_credits_for_Martin_Panter=2E?= Message-ID: <20150126081529.84261.69160@psf.io> https://hg.python.org/cpython/rev/9b0b6804528d changeset: 94291:9b0b6804528d parent: 94289:051f2a234d8a parent: 94290:05acd874ce3e user: Serhiy Storchaka date: Mon Jan 26 10:14:49 2015 +0200 summary: Add credits for Martin Panter. files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -11,7 +11,7 @@ ----------------- - Issue #21408: The default __ne__() now returns NotImplemented if __eq__() - returned NotImplemented. + returned NotImplemented. Original patch by Martin Panter. - Issue #23321: Fixed a crash in str.decode() when error handler returned replacment string longer than mailformed input data. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:15:30 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:15:30 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Add_credits_fo?= =?utf-8?q?r_Martin_Panter=2E?= Message-ID: <20150126081528.118196.4727@psf.io> https://hg.python.org/cpython/rev/05acd874ce3e changeset: 94290:05acd874ce3e branch: 3.4 parent: 94288:9a7d965ab80f user: Serhiy Storchaka date: Mon Jan 26 10:14:29 2015 +0200 summary: Add credits for Martin Panter. files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -12,7 +12,7 @@ ----------------- - Issue #21408: The default __ne__() now returns NotImplemented if __eq__() - returned NotImplemented. + returned NotImplemented. Original patch by Martin Panter. - Issue #23321: Fixed a crash in str.decode() when error handler returned replacment string longer than mailformed input data. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:28:18 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:28:18 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=237665=3A_Fixed_tests_test=5Fntpath_and_test=5Fur?= =?utf-8?q?llib2_when_ran_in_the?= Message-ID: <20150126082810.55100.42243@psf.io> https://hg.python.org/cpython/rev/1db1cd711104 changeset: 94294:1db1cd711104 parent: 94291:9b0b6804528d parent: 94293:04fa56628830 user: Serhiy Storchaka date: Mon Jan 26 10:26:29 2015 +0200 summary: Issue #7665: Fixed tests test_ntpath and test_urllib2 when ran in the directory containing a backslash. files: Lib/test/test_ntpath.py | 7 ++++--- Lib/test/test_urllib2.py | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -306,13 +306,14 @@ self.skipTest('nt module not available') def test_relpath(self): - currentdir = os.path.split(os.getcwd())[-1] tester('ntpath.relpath("a")', 'a') tester('ntpath.relpath(os.path.abspath("a"))', 'a') tester('ntpath.relpath("a/b")', 'a\\b') tester('ntpath.relpath("../a/b")', '..\\a\\b') - tester('ntpath.relpath("a", "../b")', '..\\'+currentdir+'\\a') - tester('ntpath.relpath("a/b", "../c")', '..\\'+currentdir+'\\a\\b') + with support.temp_cwd(support.TESTFN) as cwd_dir: + currentdir = os.path.basename(cwd_dir) + tester('ntpath.relpath("a", "../b")', '..\\'+currentdir+'\\a') + tester('ntpath.relpath("a/b", "../c")', '..\\'+currentdir+'\\a\\b') tester('ntpath.relpath("a", "b/c")', '..\\..\\a') tester('ntpath.relpath("c:/foo/bar/bat", "c:/x/y")', '..\\..\\foo\\bar\\bat') tester('ntpath.relpath("//conky/mountpoint/a", "//conky/mountpoint/b/c")', '..\\..\\a') diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -44,7 +44,7 @@ self.assertRaises(ValueError, urllib.request.urlopen, 'bogus url') # XXX Name hacking to get this to work on Windows. - fname = os.path.abspath(urllib.request.__file__).replace('\\', '/') + fname = os.path.abspath(urllib.request.__file__).replace(os.sep, '/') if os.name == 'nt': file_url = "file:///%s" % fname -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:28:18 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:28:18 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzc2NjU6?= =?utf-8?q?_Fixed_tests_test=5Fntpath_and_test=5Furllib2_when_ran_in_the?= Message-ID: <20150126082810.87137.2161@psf.io> https://hg.python.org/cpython/rev/04fa56628830 changeset: 94293:04fa56628830 branch: 3.4 parent: 94290:05acd874ce3e user: Serhiy Storchaka date: Mon Jan 26 10:26:14 2015 +0200 summary: Issue #7665: Fixed tests test_ntpath and test_urllib2 when ran in the directory containing a backslash. files: Lib/test/test_ntpath.py | 7 ++++--- Lib/test/test_urllib2.py | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -306,13 +306,14 @@ self.skipTest('nt module not available') def test_relpath(self): - currentdir = os.path.split(os.getcwd())[-1] tester('ntpath.relpath("a")', 'a') tester('ntpath.relpath(os.path.abspath("a"))', 'a') tester('ntpath.relpath("a/b")', 'a\\b') tester('ntpath.relpath("../a/b")', '..\\a\\b') - tester('ntpath.relpath("a", "../b")', '..\\'+currentdir+'\\a') - tester('ntpath.relpath("a/b", "../c")', '..\\'+currentdir+'\\a\\b') + with support.temp_cwd(support.TESTFN) as cwd_dir: + currentdir = os.path.basename(cwd_dir) + tester('ntpath.relpath("a", "../b")', '..\\'+currentdir+'\\a') + tester('ntpath.relpath("a/b", "../c")', '..\\'+currentdir+'\\a\\b') tester('ntpath.relpath("a", "b/c")', '..\\..\\a') tester('ntpath.relpath("c:/foo/bar/bat", "c:/x/y")', '..\\..\\foo\\bar\\bat') tester('ntpath.relpath("//conky/mountpoint/a", "//conky/mountpoint/b/c")', '..\\..\\a') diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -44,7 +44,7 @@ self.assertRaises(ValueError, urllib.request.urlopen, 'bogus url') # XXX Name hacking to get this to work on Windows. - fname = os.path.abspath(urllib.request.__file__).replace('\\', '/') + fname = os.path.abspath(urllib.request.__file__).replace(os.sep, '/') if os.name == 'nt': file_url = "file:///%s" % fname -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:28:18 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:28:18 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzc2NjU6?= =?utf-8?q?_Fixed_tests_test=5Fntpath_and_test=5Furllib2_when_ran_in_the?= Message-ID: <20150126082810.118220.43797@psf.io> https://hg.python.org/cpython/rev/1e12c9e5bc89 changeset: 94292:1e12c9e5bc89 branch: 2.7 parent: 94207:2db41d551a4f user: Serhiy Storchaka date: Mon Jan 26 10:26:00 2015 +0200 summary: Issue #7665: Fixed tests test_ntpath and test_urllib2 when ran in the directory containing a backslash. files: Lib/test/test_ntpath.py | 7 ++++--- Lib/test/test_urllib2.py | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -281,13 +281,14 @@ tester('ntpath.abspath("C:\\")', "C:\\") def test_relpath(self): - currentdir = os.path.split(os.getcwd())[-1] tester('ntpath.relpath("a")', 'a') tester('ntpath.relpath(os.path.abspath("a"))', 'a') tester('ntpath.relpath("a/b")', 'a\\b') tester('ntpath.relpath("../a/b")', '..\\a\\b') - tester('ntpath.relpath("a", "../b")', '..\\'+currentdir+'\\a') - tester('ntpath.relpath("a/b", "../c")', '..\\'+currentdir+'\\a\\b') + with test_support.temp_cwd(test_support.TESTFN) as cwd_dir: + currentdir = os.path.basename(cwd_dir) + tester('ntpath.relpath("a", "../b")', '..\\'+currentdir+'\\a') + tester('ntpath.relpath("a/b", "../c")', '..\\'+currentdir+'\\a\\b') tester('ntpath.relpath("a", "b/c")', '..\\..\\a') tester('ntpath.relpath("//conky/mountpoint/a", "//conky/mountpoint/b/c")', '..\\..\\a') tester('ntpath.relpath("a", "a")', '.') diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -25,7 +25,7 @@ self.assertRaises(ValueError, urllib2.urlopen, 'bogus url') # XXX Name hacking to get this to work on Windows. - fname = os.path.abspath(urllib2.__file__).replace('\\', '/') + fname = os.path.abspath(urllib2.__file__).replace(os.sep, '/') # And more hacking to get it to work on MacOS. This assumes # urllib.pathname2url works, unfortunately... -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:28:18 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:28:18 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <20150126082811.118212.21447@psf.io> https://hg.python.org/cpython/rev/36ca5e765704 changeset: 94295:36ca5e765704 branch: 2.7 parent: 94292:1e12c9e5bc89 parent: 94284:9a4882b12218 user: Serhiy Storchaka date: Mon Jan 26 10:27:31 2015 +0200 summary: Merge heads files: Doc/library/argparse.rst | 9 +- Doc/library/logging.handlers.rst | 19 +- Doc/library/ssl.rst | 35 ++- Doc/library/time.rst | 13 +- Lib/httplib.py | 9 +- Lib/rfc822.py | 7 +- Lib/ssl.py | 20 +- Lib/test/test_httplib.py | 39 ++- Lib/test/test_rfc822.py | 6 + Lib/test/test_ssl.py | 64 +++- Misc/NEWS | 8 + Modules/_ssl.c | 139 +++++++- Modules/_ssl_data.h | 297 ++++++++++++++++++- Tools/ssl/make_ssl_data.py | 57 ++- setup.py | 2 +- 15 files changed, 651 insertions(+), 73 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1499,12 +1499,15 @@ * parser_class - class which will be used to create sub-parser instances, by default the class of the current parser (e.g. ArgumentParser) - * dest - name of the attribute under which sub-command name will be + * action_ - the basic type of action to be taken when this argument is + encountered at the command line + + * dest_ - name of the attribute under which sub-command name will be stored; by default None and no value is stored - * help - help for sub-parser group in help output, by default None + * help_ - help for sub-parser group in help output, by default None - * metavar - string presenting available sub-commands in help; by default it + * metavar_ - string presenting available sub-commands in help; by default it is None and presents sub-commands in form {cmd1, cmd2, ..} Some example usage:: diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -185,15 +185,16 @@ You can use the *maxBytes* and *backupCount* values to allow the file to :dfn:`rollover` at a predetermined size. When the size is about to be exceeded, the file is closed and a new file is silently opened for output. Rollover occurs - whenever the current log file is nearly *maxBytes* in length; if *maxBytes* is - zero, rollover never occurs. If *backupCount* is non-zero, the system will save - old log files by appending the extensions '.1', '.2' etc., to the filename. For - example, with a *backupCount* of 5 and a base file name of :file:`app.log`, you - would get :file:`app.log`, :file:`app.log.1`, :file:`app.log.2`, up to - :file:`app.log.5`. The file being written to is always :file:`app.log`. When - this file is filled, it is closed and renamed to :file:`app.log.1`, and if files - :file:`app.log.1`, :file:`app.log.2`, etc. exist, then they are renamed to - :file:`app.log.2`, :file:`app.log.3` etc. respectively. + whenever the current log file is nearly *maxBytes* in length; if either of + *maxBytes* or *backupCount* is zero, rollover never occurs. If *backupCount* + is non-zero, the system will save old log files by appending the extensions + '.1', '.2' etc., to the filename. For example, with a *backupCount* of 5 and + a base file name of :file:`app.log`, you would get :file:`app.log`, + :file:`app.log.1`, :file:`app.log.2`, up to :file:`app.log.5`. The file being + written to is always :file:`app.log`. When this file is filled, it is closed + and renamed to :file:`app.log.1`, and if files :file:`app.log.1`, + :file:`app.log.2`, etc. exist, then they are renamed to :file:`app.log.2`, + :file:`app.log.3` etc. respectively. .. versionchanged:: 2.6 *delay* was added. diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -638,6 +638,13 @@ .. versionadded:: 2.7.9 +.. data:: HAS_ALPN + + Whether the OpenSSL library has built-in support for the *Application-Layer + Protocol Negotiation* TLS extension as described in :rfc:`7301`. + + .. versionadded:: 2.7.10 + .. data:: HAS_ECDH Whether the OpenSSL library has built-in support for Elliptic Curve-based @@ -864,9 +871,19 @@ .. versionadded:: 2.7.9 +.. method:: SSLSocket.selected_alpn_protocol() + + Return the protocol that was selected during the TLS handshake. If + :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does + not support ALPN, if this socket does not support any of the client's + proposed protocols, or if the handshake has not happened yet, ``None`` is + returned. + + .. versionadded:: 2.7.10 + .. method:: SSLSocket.selected_npn_protocol() - Returns the higher-level protocol that was selected during the TLS/SSL + Return the higher-level protocol that was selected during the TLS/SSL handshake. If :meth:`SSLContext.set_npn_protocols` was not called, or if the other party does not support NPN, or if the handshake has not yet happened, this will return ``None``. @@ -1034,6 +1051,20 @@ when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_alpn_protocols(protocols) + + Specify which protocols the socket should advertise during the SSL/TLS + handshake. It should be a list of ASCII strings, like ``['http/1.1', + 'spdy/2']``, ordered by preference. The selection of a protocol will happen + during the handshake, and will play out according to :rfc:`7301`. After a + successful handshake, the :meth:`SSLSocket.selected_alpn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is + False. + + .. versionadded:: 2.7.10 + .. method:: SSLContext.set_npn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS @@ -1072,7 +1103,7 @@ Due to the early negotiation phase of the TLS connection, only limited methods and attributes are usable like - :meth:`SSLSocket.selected_npn_protocol` and :attr:`SSLSocket.context`. + :meth:`SSLSocket.selected_alpn_protocol` and :attr:`SSLSocket.context`. :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.cipher` and :meth:`SSLSocket.compress` methods require that the TLS connection has progressed beyond the TLS Client Hello and therefore diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -222,12 +222,13 @@ .. function:: sleep(secs) - Suspend execution for the given number of seconds. The argument may be a - floating point number to indicate a more precise sleep time. The actual - suspension time may be less than that requested because any caught signal will - terminate the :func:`sleep` following execution of that signal's catching - routine. Also, the suspension time may be longer than requested by an arbitrary - amount because of the scheduling of other activity in the system. + Suspend execution of the current thread for the given number of seconds. + The argument may be a floating point number to indicate a more precise sleep + time. The actual suspension time may be less than that requested because any + caught signal will terminate the :func:`sleep` following execution of that + signal's catching routine. Also, the suspension time may be longer than + requested by an arbitrary amount because of the scheduling of other activity + in the system. .. function:: strftime(format[, t]) diff --git a/Lib/httplib.py b/Lib/httplib.py --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -313,6 +313,11 @@ hlist.append(line) self.addheader(headerseen, line[len(headerseen)+1:].strip()) continue + elif headerseen is not None: + # An empty header name. These aren't allowed in HTTP, but it's + # probably a benign mistake. Don't add the header, just keep + # going. + continue else: # It's not a header line; throw it back and stop here. if not self.dict: @@ -723,7 +728,7 @@ endpoint passed to set_tunnel. This is done by sending a HTTP CONNECT request to the proxy server when the connection is established. - This method must be called before the HTML connection has been + This method must be called before the HTTP connection has been established. The headers argument should be a mapping of extra HTTP headers @@ -1129,7 +1134,7 @@ "Accept arguments to set the host/port, since the superclass doesn't." if host is not None: - self._conn._set_hostport(host, port) + (self._conn.host, self._conn.port) = self._conn._get_hostport(host, port) self._conn.connect() def getfile(self): diff --git a/Lib/rfc822.py b/Lib/rfc822.py --- a/Lib/rfc822.py +++ b/Lib/rfc822.py @@ -179,6 +179,11 @@ lst.append(line) self.dict[headerseen] = line[len(headerseen)+1:].strip() continue + elif headerseen is not None: + # An empty header name. These aren't allowed in HTTP, but it's + # probably a benign mistake. Don't add the header, just keep + # going. + continue else: # It's not a header line; throw it back and stop here. if not self.dict: @@ -202,7 +207,7 @@ data in RFC 2822-like formats with special header formats. """ i = line.find(':') - if i > 0: + if i > -1: return line[:i].lower() return None diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -123,7 +123,7 @@ _import_symbols('SSL_ERROR_') _import_symbols('PROTOCOL_') -from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN +from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN, HAS_ALPN from _ssl import _OPENSSL_API_VERSION @@ -365,6 +365,17 @@ self._set_npn_protocols(protos) + def set_alpn_protocols(self, alpn_protocols): + protos = bytearray() + for protocol in alpn_protocols: + b = protocol.encode('ascii') + if len(b) == 0 or len(b) > 255: + raise SSLError('ALPN protocols must be 1 to 255 in length') + protos.append(len(b)) + protos.extend(b) + + self._set_alpn_protocols(protos) + def _load_windows_store_certs(self, storename, purpose): certs = bytearray() for cert, encoding, trust in enum_certificates(storename): @@ -647,6 +658,13 @@ else: return self._sslobj.selected_npn_protocol() + def selected_alpn_protocol(self): + self._checkClosed() + if not self._sslobj or not _ssl.HAS_ALPN: + return None + else: + return self._sslobj.selected_alpn_protocol() + def cipher(self): self._checkClosed() if not self._sslobj: diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -164,6 +164,16 @@ conn.request('GET', '/foo') self.assertTrue(sock.data.startswith(expected)) + def test_malformed_headers_coped_with(self): + # Issue 19996 + body = "HTTP/1.1 200 OK\r\nFirst: val\r\n: nval\r\nSecond: val\r\n\r\n" + sock = FakeSocket(body) + resp = httplib.HTTPResponse(sock) + resp.begin() + + self.assertEqual(resp.getheader('First'), 'val') + self.assertEqual(resp.getheader('Second'), 'val') + class BasicTest(TestCase): def test_status_lines(self): @@ -461,7 +471,11 @@ self.assertEqual(httplib.responses[httplib.NOT_FOUND], "Not Found") -class SourceAddressTest(TestCase): +class TestServerMixin: + """A limited socket server mixin. + + This is used by test cases for testing http connection end points. + """ def setUp(self): self.serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.port = test_support.bind_port(self.serv) @@ -476,6 +490,7 @@ self.serv.close() self.serv = None +class SourceAddressTest(TestServerMixin, TestCase): def testHTTPConnectionSourceAddress(self): self.conn = httplib.HTTPConnection(HOST, self.port, source_address=('', self.source_port)) @@ -492,6 +507,24 @@ # for an ssl_wrapped connect() to actually return from. +class HTTPTest(TestServerMixin, TestCase): + def testHTTPConnection(self): + self.conn = httplib.HTTP(host=HOST, port=self.port, strict=None) + self.conn.connect() + self.assertEqual(self.conn._conn.host, HOST) + self.assertEqual(self.conn._conn.port, self.port) + + def testHTTPWithConnectHostPort(self): + testhost = 'unreachable.test.domain' + testport = '80' + self.conn = httplib.HTTP(host=testhost, port=testport) + self.conn.connect(host=HOST, port=self.port) + self.assertNotEqual(self.conn._conn.host, testhost) + self.assertNotEqual(self.conn._conn.port, testport) + self.assertEqual(self.conn._conn.host, HOST) + self.assertEqual(self.conn._conn.port, self.port) + + class TimeoutTest(TestCase): PORT = None @@ -537,6 +570,7 @@ self.assertEqual(httpConn.sock.gettimeout(), 30) httpConn.close() + class HTTPSTest(TestCase): def setUp(self): @@ -713,7 +747,8 @@ @test_support.reap_threads def test_main(verbose=None): test_support.run_unittest(HeaderTests, OfflineTest, BasicTest, TimeoutTest, - HTTPSTest, SourceAddressTest, TunnelTests) + HTTPTest, HTTPSTest, SourceAddressTest, + TunnelTests) if __name__ == '__main__': test_main() diff --git a/Lib/test/test_rfc822.py b/Lib/test/test_rfc822.py --- a/Lib/test/test_rfc822.py +++ b/Lib/test/test_rfc822.py @@ -248,6 +248,12 @@ eq(rfc822.quote('foo\\wacky"name'), 'foo\\\\wacky\\"name') eq(rfc822.unquote('"foo\\\\wacky\\"name"'), 'foo\\wacky"name') + def test_invalid_headers(self): + eq = self.assertEqual + msg = self.create_message("First: val\n: otherval\nSecond: val2\n") + eq(msg.getheader('First'), 'val') + eq(msg.getheader('Second'), 'val2') + def test_main(): test_support.run_unittest(MessageTestCase) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1569,7 +1569,8 @@ try: self.sslconn = self.server.context.wrap_socket( self.sock, server_side=True) - self.server.selected_protocols.append(self.sslconn.selected_npn_protocol()) + self.server.selected_npn_protocols.append(self.sslconn.selected_npn_protocol()) + self.server.selected_alpn_protocols.append(self.sslconn.selected_alpn_protocol()) except socket.error as e: # We treat ConnectionResetError as though it were an # SSLError - OpenSSL on Ubuntu abruptly closes the @@ -1678,7 +1679,8 @@ def __init__(self, certificate=None, ssl_version=None, certreqs=None, cacerts=None, chatty=True, connectionchatty=False, starttls_server=False, - npn_protocols=None, ciphers=None, context=None): + npn_protocols=None, alpn_protocols=None, + ciphers=None, context=None): if context: self.context = context else: @@ -1693,6 +1695,8 @@ self.context.load_cert_chain(certificate) if npn_protocols: self.context.set_npn_protocols(npn_protocols) + if alpn_protocols: + self.context.set_alpn_protocols(alpn_protocols) if ciphers: self.context.set_ciphers(ciphers) self.chatty = chatty @@ -1702,7 +1706,8 @@ self.port = support.bind_port(self.sock) self.flag = None self.active = False - self.selected_protocols = [] + self.selected_npn_protocols = [] + self.selected_alpn_protocols = [] self.conn_errors = [] threading.Thread.__init__(self) self.daemon = True @@ -1927,11 +1932,13 @@ 'compression': s.compression(), 'cipher': s.cipher(), 'peercert': s.getpeercert(), + 'client_alpn_protocol': s.selected_alpn_protocol(), 'client_npn_protocol': s.selected_npn_protocol(), 'version': s.version(), }) s.close() - stats['server_npn_protocols'] = server.selected_protocols + stats['server_alpn_protocols'] = server.selected_alpn_protocols + stats['server_npn_protocols'] = server.selected_npn_protocols return stats def try_protocol_combo(server_protocol, client_protocol, expect_success, @@ -2787,6 +2794,55 @@ if "ADH" not in parts and "EDH" not in parts and "DHE" not in parts: self.fail("Non-DH cipher: " + cipher[0]) + def test_selected_alpn_protocol(self): + # selected_alpn_protocol() is None unless ALPN is used. + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.load_cert_chain(CERTFILE) + stats = server_params_test(context, context, + chatty=True, connectionchatty=True) + self.assertIs(stats['client_alpn_protocol'], None) + + @unittest.skipUnless(ssl.HAS_ALPN, "ALPN support required") + def test_selected_alpn_protocol_if_server_uses_alpn(self): + # selected_alpn_protocol() is None unless ALPN is used by the client. + client_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + client_context.load_verify_locations(CERTFILE) + server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + server_context.load_cert_chain(CERTFILE) + server_context.set_alpn_protocols(['foo', 'bar']) + stats = server_params_test(client_context, server_context, + chatty=True, connectionchatty=True) + self.assertIs(stats['client_alpn_protocol'], None) + + @unittest.skipUnless(ssl.HAS_ALPN, "ALPN support needed for this test") + def test_alpn_protocols(self): + server_protocols = ['foo', 'bar', 'milkshake'] + protocol_tests = [ + (['foo', 'bar'], 'foo'), + (['bar', 'foo'], 'foo'), + (['milkshake'], 'milkshake'), + (['http/3.0', 'http/4.0'], None) + ] + for client_protocols, expected in protocol_tests: + server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + server_context.load_cert_chain(CERTFILE) + server_context.set_alpn_protocols(server_protocols) + client_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + client_context.load_cert_chain(CERTFILE) + client_context.set_alpn_protocols(client_protocols) + stats = server_params_test(client_context, server_context, + chatty=True, connectionchatty=True) + + msg = "failed trying %s (s) and %s (c).\n" \ + "was expecting %s, but got %%s from the %%s" \ + % (str(server_protocols), str(client_protocols), + str(expected)) + client_result = stats['client_alpn_protocol'] + self.assertEqual(client_result, expected, msg % (client_result, "client")) + server_result = stats['server_alpn_protocols'][-1] \ + if len(stats['server_alpn_protocols']) else 'nothing' + self.assertEqual(server_result, expected, msg % (server_result, "server")) + def test_selected_npn_protocol(self): # selected_npn_protocol() is None unless NPN is used context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,14 @@ Library ------- +- Issue #19996: Make :mod:`httplib` ignore headers with no name rather than + assuming the body has started. + +- Issue #20188: Support Application-Layer Protocol Negotiation (ALPN) in the ssl + module. + +- Issue #23248: Update ssl error codes from latest OpenSSL git master. + - Issue #23098: 64-bit dev_t is now supported in the os module. - Issue #23063: In the disutils' check command, fix parsing of reST with code or diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -105,6 +105,11 @@ # define HAVE_SNI 0 #endif +/* ALPN added in OpenSSL 1.0.2 */ +#if OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) +# define HAVE_ALPN +#endif + enum py_ssl_error { /* these mirror ssl.h */ PY_SSL_ERROR_NONE, @@ -205,9 +210,13 @@ PyObject_HEAD SSL_CTX *ctx; #ifdef OPENSSL_NPN_NEGOTIATED - char *npn_protocols; + unsigned char *npn_protocols; int npn_protocols_len; #endif +#ifdef HAVE_ALPN + unsigned char *alpn_protocols; + int alpn_protocols_len; +#endif #ifndef OPENSSL_NO_TLSEXT PyObject *set_hostname; #endif @@ -1408,7 +1417,20 @@ if (out == NULL) Py_RETURN_NONE; - return PyUnicode_FromStringAndSize((char *) out, outlen); + return PyString_FromStringAndSize((char *)out, outlen); +} +#endif + +#ifdef HAVE_ALPN +static PyObject *PySSL_selected_alpn_protocol(PySSLSocket *self) { + const unsigned char *out; + unsigned int outlen; + + SSL_get0_alpn_selected(self->ssl, &out, &outlen); + + if (out == NULL) + Py_RETURN_NONE; + return PyString_FromStringAndSize((char *)out, outlen); } #endif @@ -1925,6 +1947,9 @@ #ifdef OPENSSL_NPN_NEGOTIATED {"selected_npn_protocol", (PyCFunction)PySSL_selected_npn_protocol, METH_NOARGS}, #endif +#ifdef HAVE_ALPN + {"selected_alpn_protocol", (PyCFunction)PySSL_selected_alpn_protocol, METH_NOARGS}, +#endif {"compression", (PyCFunction)PySSL_compression, METH_NOARGS}, {"shutdown", (PyCFunction)PySSL_SSLshutdown, METH_NOARGS, PySSL_SSLshutdown_doc}, @@ -2032,6 +2057,9 @@ #ifdef OPENSSL_NPN_NEGOTIATED self->npn_protocols = NULL; #endif +#ifdef HAVE_ALPN + self->alpn_protocols = NULL; +#endif #ifndef OPENSSL_NO_TLSEXT self->set_hostname = NULL; #endif @@ -2091,7 +2119,10 @@ context_clear(self); SSL_CTX_free(self->ctx); #ifdef OPENSSL_NPN_NEGOTIATED - PyMem_Free(self->npn_protocols); + PyMem_FREE(self->npn_protocols); +#endif +#ifdef HAVE_ALPN + PyMem_FREE(self->alpn_protocols); #endif Py_TYPE(self)->tp_free(self); } @@ -2117,6 +2148,30 @@ Py_RETURN_NONE; } +static int +do_protocol_selection(int alpn, unsigned char **out, unsigned char *outlen, + const unsigned char *server_protocols, unsigned int server_protocols_len, + const unsigned char *client_protocols, unsigned int client_protocols_len) +{ + int ret; + if (client_protocols == NULL) { + client_protocols = (unsigned char *)""; + client_protocols_len = 0; + } + if (server_protocols == NULL) { + server_protocols = (unsigned char *)""; + server_protocols_len = 0; + } + + ret = SSL_select_next_proto(out, outlen, + server_protocols, server_protocols_len, + client_protocols, client_protocols_len); + if (alpn && ret != OPENSSL_NPN_NEGOTIATED) + return SSL_TLSEXT_ERR_NOACK; + + return SSL_TLSEXT_ERR_OK; +} + #ifdef OPENSSL_NPN_NEGOTIATED /* this callback gets passed to SSL_CTX_set_next_protos_advertise_cb */ static int @@ -2127,10 +2182,10 @@ PySSLContext *ssl_ctx = (PySSLContext *) args; if (ssl_ctx->npn_protocols == NULL) { - *data = (unsigned char *) ""; + *data = (unsigned char *)""; *len = 0; } else { - *data = (unsigned char *) ssl_ctx->npn_protocols; + *data = ssl_ctx->npn_protocols; *len = ssl_ctx->npn_protocols_len; } @@ -2143,23 +2198,9 @@ const unsigned char *server, unsigned int server_len, void *args) { - PySSLContext *ssl_ctx = (PySSLContext *) args; - - unsigned char *client = (unsigned char *) ssl_ctx->npn_protocols; - int client_len; - - if (client == NULL) { - client = (unsigned char *) ""; - client_len = 0; - } else { - client_len = ssl_ctx->npn_protocols_len; - } - - SSL_select_next_proto(out, outlen, - server, server_len, - client, client_len); - - return SSL_TLSEXT_ERR_OK; + PySSLContext *ctx = (PySSLContext *)args; + return do_protocol_selection(0, out, outlen, server, server_len, + ctx->npn_protocols, ctx->npn_protocols_len); } #endif @@ -2202,6 +2243,50 @@ #endif } +#ifdef HAVE_ALPN +static int +_selectALPN_cb(SSL *s, + const unsigned char **out, unsigned char *outlen, + const unsigned char *client_protocols, unsigned int client_protocols_len, + void *args) +{ + PySSLContext *ctx = (PySSLContext *)args; + return do_protocol_selection(1, (unsigned char **)out, outlen, + ctx->alpn_protocols, ctx->alpn_protocols_len, + client_protocols, client_protocols_len); +} +#endif + +static PyObject * +_set_alpn_protocols(PySSLContext *self, PyObject *args) +{ +#ifdef HAVE_ALPN + Py_buffer protos; + + if (!PyArg_ParseTuple(args, "s*:set_npn_protocols", &protos)) + return NULL; + + PyMem_FREE(self->alpn_protocols); + self->alpn_protocols = PyMem_Malloc(protos.len); + if (!self->alpn_protocols) + return PyErr_NoMemory(); + memcpy(self->alpn_protocols, protos.buf, protos.len); + self->alpn_protocols_len = protos.len; + PyBuffer_Release(&protos); + + if (SSL_CTX_set_alpn_protos(self->ctx, self->alpn_protocols, self->alpn_protocols_len)) + return PyErr_NoMemory(); + SSL_CTX_set_alpn_select_cb(self->ctx, _selectALPN_cb, self); + + PyBuffer_Release(&protos); + Py_RETURN_NONE; +#else + PyErr_SetString(PyExc_NotImplementedError, + "The ALPN extension requires OpenSSL 1.0.2 or later."); + return NULL; +#endif +} + static PyObject * get_verify_mode(PySSLContext *self, void *c) { @@ -3188,6 +3273,8 @@ METH_VARARGS | METH_KEYWORDS, NULL}, {"set_ciphers", (PyCFunction) set_ciphers, METH_VARARGS, NULL}, + {"_set_alpn_protocols", (PyCFunction) _set_alpn_protocols, + METH_VARARGS, NULL}, {"_set_npn_protocols", (PyCFunction) _set_npn_protocols, METH_VARARGS, NULL}, {"load_cert_chain", (PyCFunction) load_cert_chain, @@ -4100,6 +4187,14 @@ Py_INCREF(r); PyModule_AddObject(m, "HAS_NPN", r); +#ifdef HAVE_ALPN + r = Py_True; +#else + r = Py_False; +#endif + Py_INCREF(r); + PyModule_AddObject(m, "HAS_ALPN", r); + /* Mappings for error codes */ err_codes_to_names = PyDict_New(); err_names_to_codes = PyDict_New(); diff --git a/Modules/_ssl_data.h b/Modules/_ssl_data.h --- a/Modules/_ssl_data.h +++ b/Modules/_ssl_data.h @@ -1,5 +1,5 @@ /* File generated by Tools/ssl/make_ssl_data.py */ -/* Generated on 2012-05-16T23:56:40.981382 */ +/* Generated on 2015-01-17T20:33:43.377453 */ static struct py_ssl_library_code library_codes[] = { {"PEM", ERR_LIB_PEM}, @@ -179,6 +179,11 @@ #else {"BAD_CHECKSUM", ERR_LIB_SSL, 104}, #endif + #ifdef SSL_R_BAD_DATA + {"BAD_DATA", ERR_LIB_SSL, SSL_R_BAD_DATA}, + #else + {"BAD_DATA", ERR_LIB_SSL, 390}, + #endif #ifdef SSL_R_BAD_DATA_RETURNED_BY_CALLBACK {"BAD_DATA_RETURNED_BY_CALLBACK", ERR_LIB_SSL, SSL_R_BAD_DATA_RETURNED_BY_CALLBACK}, #else @@ -309,6 +314,46 @@ #else {"BAD_SIGNATURE", ERR_LIB_SSL, 123}, #endif + #ifdef SSL_R_BAD_SRP_A_LENGTH + {"BAD_SRP_A_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_A_LENGTH}, + #else + {"BAD_SRP_A_LENGTH", ERR_LIB_SSL, 347}, + #endif + #ifdef SSL_R_BAD_SRP_B_LENGTH + {"BAD_SRP_B_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_B_LENGTH}, + #else + {"BAD_SRP_B_LENGTH", ERR_LIB_SSL, 348}, + #endif + #ifdef SSL_R_BAD_SRP_G_LENGTH + {"BAD_SRP_G_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_G_LENGTH}, + #else + {"BAD_SRP_G_LENGTH", ERR_LIB_SSL, 349}, + #endif + #ifdef SSL_R_BAD_SRP_N_LENGTH + {"BAD_SRP_N_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_N_LENGTH}, + #else + {"BAD_SRP_N_LENGTH", ERR_LIB_SSL, 350}, + #endif + #ifdef SSL_R_BAD_SRP_PARAMETERS + {"BAD_SRP_PARAMETERS", ERR_LIB_SSL, SSL_R_BAD_SRP_PARAMETERS}, + #else + {"BAD_SRP_PARAMETERS", ERR_LIB_SSL, 371}, + #endif + #ifdef SSL_R_BAD_SRP_S_LENGTH + {"BAD_SRP_S_LENGTH", ERR_LIB_SSL, SSL_R_BAD_SRP_S_LENGTH}, + #else + {"BAD_SRP_S_LENGTH", ERR_LIB_SSL, 351}, + #endif + #ifdef SSL_R_BAD_SRTP_MKI_VALUE + {"BAD_SRTP_MKI_VALUE", ERR_LIB_SSL, SSL_R_BAD_SRTP_MKI_VALUE}, + #else + {"BAD_SRTP_MKI_VALUE", ERR_LIB_SSL, 352}, + #endif + #ifdef SSL_R_BAD_SRTP_PROTECTION_PROFILE_LIST + {"BAD_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, SSL_R_BAD_SRTP_PROTECTION_PROFILE_LIST}, + #else + {"BAD_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, 353}, + #endif #ifdef SSL_R_BAD_SSL_FILETYPE {"BAD_SSL_FILETYPE", ERR_LIB_SSL, SSL_R_BAD_SSL_FILETYPE}, #else @@ -324,6 +369,11 @@ #else {"BAD_STATE", ERR_LIB_SSL, 126}, #endif + #ifdef SSL_R_BAD_VALUE + {"BAD_VALUE", ERR_LIB_SSL, SSL_R_BAD_VALUE}, + #else + {"BAD_VALUE", ERR_LIB_SSL, 384}, + #endif #ifdef SSL_R_BAD_WRITE_RETRY {"BAD_WRITE_RETRY", ERR_LIB_SSL, SSL_R_BAD_WRITE_RETRY}, #else @@ -354,6 +404,16 @@ #else {"CA_DN_TOO_LONG", ERR_LIB_SSL, 132}, #endif + #ifdef SSL_R_CA_KEY_TOO_SMALL + {"CA_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_CA_KEY_TOO_SMALL}, + #else + {"CA_KEY_TOO_SMALL", ERR_LIB_SSL, 397}, + #endif + #ifdef SSL_R_CA_MD_TOO_WEAK + {"CA_MD_TOO_WEAK", ERR_LIB_SSL, SSL_R_CA_MD_TOO_WEAK}, + #else + {"CA_MD_TOO_WEAK", ERR_LIB_SSL, 398}, + #endif #ifdef SSL_R_CCS_RECEIVED_EARLY {"CCS_RECEIVED_EARLY", ERR_LIB_SSL, SSL_R_CCS_RECEIVED_EARLY}, #else @@ -364,6 +424,11 @@ #else {"CERTIFICATE_VERIFY_FAILED", ERR_LIB_SSL, 134}, #endif + #ifdef SSL_R_CERT_CB_ERROR + {"CERT_CB_ERROR", ERR_LIB_SSL, SSL_R_CERT_CB_ERROR}, + #else + {"CERT_CB_ERROR", ERR_LIB_SSL, 377}, + #endif #ifdef SSL_R_CERT_LENGTH_MISMATCH {"CERT_LENGTH_MISMATCH", ERR_LIB_SSL, SSL_R_CERT_LENGTH_MISMATCH}, #else @@ -454,6 +519,11 @@ #else {"DECRYPTION_FAILED_OR_BAD_RECORD_MAC", ERR_LIB_SSL, 281}, #endif + #ifdef SSL_R_DH_KEY_TOO_SMALL + {"DH_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_DH_KEY_TOO_SMALL}, + #else + {"DH_KEY_TOO_SMALL", ERR_LIB_SSL, 394}, + #endif #ifdef SSL_R_DH_PUBLIC_VALUE_LENGTH_IS_WRONG {"DH_PUBLIC_VALUE_LENGTH_IS_WRONG", ERR_LIB_SSL, SSL_R_DH_PUBLIC_VALUE_LENGTH_IS_WRONG}, #else @@ -494,11 +564,26 @@ #else {"ECC_CERT_SHOULD_HAVE_SHA1_SIGNATURE", ERR_LIB_SSL, 323}, #endif + #ifdef SSL_R_ECDH_REQUIRED_FOR_SUITEB_MODE + {"ECDH_REQUIRED_FOR_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ECDH_REQUIRED_FOR_SUITEB_MODE}, + #else + {"ECDH_REQUIRED_FOR_SUITEB_MODE", ERR_LIB_SSL, 374}, + #endif #ifdef SSL_R_ECGROUP_TOO_LARGE_FOR_CIPHER {"ECGROUP_TOO_LARGE_FOR_CIPHER", ERR_LIB_SSL, SSL_R_ECGROUP_TOO_LARGE_FOR_CIPHER}, #else {"ECGROUP_TOO_LARGE_FOR_CIPHER", ERR_LIB_SSL, 310}, #endif + #ifdef SSL_R_EE_KEY_TOO_SMALL + {"EE_KEY_TOO_SMALL", ERR_LIB_SSL, SSL_R_EE_KEY_TOO_SMALL}, + #else + {"EE_KEY_TOO_SMALL", ERR_LIB_SSL, 399}, + #endif + #ifdef SSL_R_EMPTY_SRTP_PROTECTION_PROFILE_LIST + {"EMPTY_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, SSL_R_EMPTY_SRTP_PROTECTION_PROFILE_LIST}, + #else + {"EMPTY_SRTP_PROTECTION_PROFILE_LIST", ERR_LIB_SSL, 354}, + #endif #ifdef SSL_R_ENCRYPTED_LENGTH_TOO_LONG {"ENCRYPTED_LENGTH_TOO_LONG", ERR_LIB_SSL, SSL_R_ENCRYPTED_LENGTH_TOO_LONG}, #else @@ -529,6 +614,16 @@ #else {"GOT_A_FIN_BEFORE_A_CCS", ERR_LIB_SSL, 154}, #endif + #ifdef SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS + {"GOT_NEXT_PROTO_BEFORE_A_CCS", ERR_LIB_SSL, SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS}, + #else + {"GOT_NEXT_PROTO_BEFORE_A_CCS", ERR_LIB_SSL, 355}, + #endif + #ifdef SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION + {"GOT_NEXT_PROTO_WITHOUT_EXTENSION", ERR_LIB_SSL, SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION}, + #else + {"GOT_NEXT_PROTO_WITHOUT_EXTENSION", ERR_LIB_SSL, 356}, + #endif #ifdef SSL_R_HTTPS_PROXY_REQUEST {"HTTPS_PROXY_REQUEST", ERR_LIB_SSL, SSL_R_HTTPS_PROXY_REQUEST}, #else @@ -544,6 +639,16 @@ #else {"ILLEGAL_PADDING", ERR_LIB_SSL, 283}, #endif + #ifdef SSL_R_ILLEGAL_SUITEB_DIGEST + {"ILLEGAL_SUITEB_DIGEST", ERR_LIB_SSL, SSL_R_ILLEGAL_SUITEB_DIGEST}, + #else + {"ILLEGAL_SUITEB_DIGEST", ERR_LIB_SSL, 380}, + #endif + #ifdef SSL_R_INAPPROPRIATE_FALLBACK + {"INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, SSL_R_INAPPROPRIATE_FALLBACK}, + #else + {"INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, 373}, + #endif #ifdef SSL_R_INCONSISTENT_COMPRESSION {"INCONSISTENT_COMPRESSION", ERR_LIB_SSL, SSL_R_INCONSISTENT_COMPRESSION}, #else @@ -564,11 +669,26 @@ #else {"INVALID_COMPRESSION_ALGORITHM", ERR_LIB_SSL, 341}, #endif + #ifdef SSL_R_INVALID_NULL_CMD_NAME + {"INVALID_NULL_CMD_NAME", ERR_LIB_SSL, SSL_R_INVALID_NULL_CMD_NAME}, + #else + {"INVALID_NULL_CMD_NAME", ERR_LIB_SSL, 385}, + #endif #ifdef SSL_R_INVALID_PURPOSE {"INVALID_PURPOSE", ERR_LIB_SSL, SSL_R_INVALID_PURPOSE}, #else {"INVALID_PURPOSE", ERR_LIB_SSL, 278}, #endif + #ifdef SSL_R_INVALID_SERVERINFO_DATA + {"INVALID_SERVERINFO_DATA", ERR_LIB_SSL, SSL_R_INVALID_SERVERINFO_DATA}, + #else + {"INVALID_SERVERINFO_DATA", ERR_LIB_SSL, 388}, + #endif + #ifdef SSL_R_INVALID_SRP_USERNAME + {"INVALID_SRP_USERNAME", ERR_LIB_SSL, SSL_R_INVALID_SRP_USERNAME}, + #else + {"INVALID_SRP_USERNAME", ERR_LIB_SSL, 357}, + #endif #ifdef SSL_R_INVALID_STATUS_RESPONSE {"INVALID_STATUS_RESPONSE", ERR_LIB_SSL, SSL_R_INVALID_STATUS_RESPONSE}, #else @@ -689,6 +809,16 @@ #else {"MISSING_DSA_SIGNING_CERT", ERR_LIB_SSL, 165}, #endif + #ifdef SSL_R_MISSING_ECDH_CERT + {"MISSING_ECDH_CERT", ERR_LIB_SSL, SSL_R_MISSING_ECDH_CERT}, + #else + {"MISSING_ECDH_CERT", ERR_LIB_SSL, 382}, + #endif + #ifdef SSL_R_MISSING_ECDSA_SIGNING_CERT + {"MISSING_ECDSA_SIGNING_CERT", ERR_LIB_SSL, SSL_R_MISSING_ECDSA_SIGNING_CERT}, + #else + {"MISSING_ECDSA_SIGNING_CERT", ERR_LIB_SSL, 381}, + #endif #ifdef SSL_R_MISSING_EXPORT_TMP_DH_KEY {"MISSING_EXPORT_TMP_DH_KEY", ERR_LIB_SSL, SSL_R_MISSING_EXPORT_TMP_DH_KEY}, #else @@ -714,6 +844,11 @@ #else {"MISSING_RSA_SIGNING_CERT", ERR_LIB_SSL, 170}, #endif + #ifdef SSL_R_MISSING_SRP_PARAM + {"MISSING_SRP_PARAM", ERR_LIB_SSL, SSL_R_MISSING_SRP_PARAM}, + #else + {"MISSING_SRP_PARAM", ERR_LIB_SSL, 358}, + #endif #ifdef SSL_R_MISSING_TMP_DH_KEY {"MISSING_TMP_DH_KEY", ERR_LIB_SSL, SSL_R_MISSING_TMP_DH_KEY}, #else @@ -739,6 +874,11 @@ #else {"MISSING_VERIFY_MESSAGE", ERR_LIB_SSL, 174}, #endif + #ifdef SSL_R_MULTIPLE_SGC_RESTARTS + {"MULTIPLE_SGC_RESTARTS", ERR_LIB_SSL, SSL_R_MULTIPLE_SGC_RESTARTS}, + #else + {"MULTIPLE_SGC_RESTARTS", ERR_LIB_SSL, 346}, + #endif #ifdef SSL_R_NON_SSLV2_INITIAL_PACKET {"NON_SSLV2_INITIAL_PACKET", ERR_LIB_SSL, SSL_R_NON_SSLV2_INITIAL_PACKET}, #else @@ -819,6 +959,11 @@ #else {"NO_METHOD_SPECIFIED", ERR_LIB_SSL, 188}, #endif + #ifdef SSL_R_NO_PEM_EXTENSIONS + {"NO_PEM_EXTENSIONS", ERR_LIB_SSL, SSL_R_NO_PEM_EXTENSIONS}, + #else + {"NO_PEM_EXTENSIONS", ERR_LIB_SSL, 389}, + #endif #ifdef SSL_R_NO_PRIVATEKEY {"NO_PRIVATEKEY", ERR_LIB_SSL, SSL_R_NO_PRIVATEKEY}, #else @@ -854,6 +999,16 @@ #else {"NO_SHARED_CIPHER", ERR_LIB_SSL, 193}, #endif + #ifdef SSL_R_NO_SHARED_SIGATURE_ALGORITHMS + {"NO_SHARED_SIGATURE_ALGORITHMS", ERR_LIB_SSL, SSL_R_NO_SHARED_SIGATURE_ALGORITHMS}, + #else + {"NO_SHARED_SIGATURE_ALGORITHMS", ERR_LIB_SSL, 376}, + #endif + #ifdef SSL_R_NO_SRTP_PROFILES + {"NO_SRTP_PROFILES", ERR_LIB_SSL, SSL_R_NO_SRTP_PROFILES}, + #else + {"NO_SRTP_PROFILES", ERR_LIB_SSL, 359}, + #endif #ifdef SSL_R_NO_VERIFY_CALLBACK {"NO_VERIFY_CALLBACK", ERR_LIB_SSL, SSL_R_NO_VERIFY_CALLBACK}, #else @@ -879,6 +1034,16 @@ #else {"OLD_SESSION_COMPRESSION_ALGORITHM_NOT_RETURNED", ERR_LIB_SSL, 344}, #endif + #ifdef SSL_R_ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE + {"ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE}, + #else + {"ONLY_DTLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, 387}, + #endif + #ifdef SSL_R_ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE + {"ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, SSL_R_ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE}, + #else + {"ONLY_TLS_1_2_ALLOWED_IN_SUITEB_MODE", ERR_LIB_SSL, 379}, + #endif #ifdef SSL_R_ONLY_TLS_ALLOWED_IN_FIPS_MODE {"ONLY_TLS_ALLOWED_IN_FIPS_MODE", ERR_LIB_SSL, SSL_R_ONLY_TLS_ALLOWED_IN_FIPS_MODE}, #else @@ -934,6 +1099,16 @@ #else {"PEER_ERROR_UNSUPPORTED_CERTIFICATE_TYPE", ERR_LIB_SSL, 204}, #endif + #ifdef SSL_R_PEM_NAME_BAD_PREFIX + {"PEM_NAME_BAD_PREFIX", ERR_LIB_SSL, SSL_R_PEM_NAME_BAD_PREFIX}, + #else + {"PEM_NAME_BAD_PREFIX", ERR_LIB_SSL, 391}, + #endif + #ifdef SSL_R_PEM_NAME_TOO_SHORT + {"PEM_NAME_TOO_SHORT", ERR_LIB_SSL, SSL_R_PEM_NAME_TOO_SHORT}, + #else + {"PEM_NAME_TOO_SHORT", ERR_LIB_SSL, 392}, + #endif #ifdef SSL_R_PRE_MAC_LENGTH_TOO_LONG {"PRE_MAC_LENGTH_TOO_LONG", ERR_LIB_SSL, SSL_R_PRE_MAC_LENGTH_TOO_LONG}, #else @@ -1069,11 +1244,36 @@ #else {"SHORT_READ", ERR_LIB_SSL, 219}, #endif + #ifdef SSL_R_SIGNATURE_ALGORITHMS_ERROR + {"SIGNATURE_ALGORITHMS_ERROR", ERR_LIB_SSL, SSL_R_SIGNATURE_ALGORITHMS_ERROR}, + #else + {"SIGNATURE_ALGORITHMS_ERROR", ERR_LIB_SSL, 360}, + #endif #ifdef SSL_R_SIGNATURE_FOR_NON_SIGNING_CERTIFICATE {"SIGNATURE_FOR_NON_SIGNING_CERTIFICATE", ERR_LIB_SSL, SSL_R_SIGNATURE_FOR_NON_SIGNING_CERTIFICATE}, #else {"SIGNATURE_FOR_NON_SIGNING_CERTIFICATE", ERR_LIB_SSL, 220}, #endif + #ifdef SSL_R_SRP_A_CALC + {"SRP_A_CALC", ERR_LIB_SSL, SSL_R_SRP_A_CALC}, + #else + {"SRP_A_CALC", ERR_LIB_SSL, 361}, + #endif + #ifdef SSL_R_SRTP_COULD_NOT_ALLOCATE_PROFILES + {"SRTP_COULD_NOT_ALLOCATE_PROFILES", ERR_LIB_SSL, SSL_R_SRTP_COULD_NOT_ALLOCATE_PROFILES}, + #else + {"SRTP_COULD_NOT_ALLOCATE_PROFILES", ERR_LIB_SSL, 362}, + #endif + #ifdef SSL_R_SRTP_PROTECTION_PROFILE_LIST_TOO_LONG + {"SRTP_PROTECTION_PROFILE_LIST_TOO_LONG", ERR_LIB_SSL, SSL_R_SRTP_PROTECTION_PROFILE_LIST_TOO_LONG}, + #else + {"SRTP_PROTECTION_PROFILE_LIST_TOO_LONG", ERR_LIB_SSL, 363}, + #endif + #ifdef SSL_R_SRTP_UNKNOWN_PROTECTION_PROFILE + {"SRTP_UNKNOWN_PROTECTION_PROFILE", ERR_LIB_SSL, SSL_R_SRTP_UNKNOWN_PROTECTION_PROFILE}, + #else + {"SRTP_UNKNOWN_PROTECTION_PROFILE", ERR_LIB_SSL, 364}, + #endif #ifdef SSL_R_SSL23_DOING_SESSION_ID_REUSE {"SSL23_DOING_SESSION_ID_REUSE", ERR_LIB_SSL, SSL_R_SSL23_DOING_SESSION_ID_REUSE}, #else @@ -1179,6 +1379,11 @@ #else {"SSL_LIBRARY_HAS_NO_CIPHERS", ERR_LIB_SSL, 230}, #endif + #ifdef SSL_R_SSL_NEGATIVE_LENGTH + {"SSL_NEGATIVE_LENGTH", ERR_LIB_SSL, SSL_R_SSL_NEGATIVE_LENGTH}, + #else + {"SSL_NEGATIVE_LENGTH", ERR_LIB_SSL, 372}, + #endif #ifdef SSL_R_SSL_SESSION_ID_CALLBACK_FAILED {"SSL_SESSION_ID_CALLBACK_FAILED", ERR_LIB_SSL, SSL_R_SSL_SESSION_ID_CALLBACK_FAILED}, #else @@ -1229,6 +1434,11 @@ #else {"TLSV1_ALERT_EXPORT_RESTRICTION", ERR_LIB_SSL, 1060}, #endif + #ifdef SSL_R_TLSV1_ALERT_INAPPROPRIATE_FALLBACK + {"TLSV1_ALERT_INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, SSL_R_TLSV1_ALERT_INAPPROPRIATE_FALLBACK}, + #else + {"TLSV1_ALERT_INAPPROPRIATE_FALLBACK", ERR_LIB_SSL, 1086}, + #endif #ifdef SSL_R_TLSV1_ALERT_INSUFFICIENT_SECURITY {"TLSV1_ALERT_INSUFFICIENT_SECURITY", ERR_LIB_SSL, SSL_R_TLSV1_ALERT_INSUFFICIENT_SECURITY}, #else @@ -1294,6 +1504,21 @@ #else {"TLS_CLIENT_CERT_REQ_WITH_ANON_CIPHER", ERR_LIB_SSL, 232}, #endif + #ifdef SSL_R_TLS_HEARTBEAT_PEER_DOESNT_ACCEPT + {"TLS_HEARTBEAT_PEER_DOESNT_ACCEPT", ERR_LIB_SSL, SSL_R_TLS_HEARTBEAT_PEER_DOESNT_ACCEPT}, + #else + {"TLS_HEARTBEAT_PEER_DOESNT_ACCEPT", ERR_LIB_SSL, 365}, + #endif + #ifdef SSL_R_TLS_HEARTBEAT_PENDING + {"TLS_HEARTBEAT_PENDING", ERR_LIB_SSL, SSL_R_TLS_HEARTBEAT_PENDING}, + #else + {"TLS_HEARTBEAT_PENDING", ERR_LIB_SSL, 366}, + #endif + #ifdef SSL_R_TLS_ILLEGAL_EXPORTER_LABEL + {"TLS_ILLEGAL_EXPORTER_LABEL", ERR_LIB_SSL, SSL_R_TLS_ILLEGAL_EXPORTER_LABEL}, + #else + {"TLS_ILLEGAL_EXPORTER_LABEL", ERR_LIB_SSL, 367}, + #endif #ifdef SSL_R_TLS_INVALID_ECPOINTFORMAT_LIST {"TLS_INVALID_ECPOINTFORMAT_LIST", ERR_LIB_SSL, SSL_R_TLS_INVALID_ECPOINTFORMAT_LIST}, #else @@ -1399,6 +1624,16 @@ #else {"UNKNOWN_CIPHER_TYPE", ERR_LIB_SSL, 249}, #endif + #ifdef SSL_R_UNKNOWN_CMD_NAME + {"UNKNOWN_CMD_NAME", ERR_LIB_SSL, SSL_R_UNKNOWN_CMD_NAME}, + #else + {"UNKNOWN_CMD_NAME", ERR_LIB_SSL, 386}, + #endif + #ifdef SSL_R_UNKNOWN_DIGEST + {"UNKNOWN_DIGEST", ERR_LIB_SSL, SSL_R_UNKNOWN_DIGEST}, + #else + {"UNKNOWN_DIGEST", ERR_LIB_SSL, 368}, + #endif #ifdef SSL_R_UNKNOWN_KEY_EXCHANGE_TYPE {"UNKNOWN_KEY_EXCHANGE_TYPE", ERR_LIB_SSL, SSL_R_UNKNOWN_KEY_EXCHANGE_TYPE}, #else @@ -1469,16 +1704,36 @@ #else {"UNSUPPORTED_STATUS_TYPE", ERR_LIB_SSL, 329}, #endif + #ifdef SSL_R_USE_SRTP_NOT_NEGOTIATED + {"USE_SRTP_NOT_NEGOTIATED", ERR_LIB_SSL, SSL_R_USE_SRTP_NOT_NEGOTIATED}, + #else + {"USE_SRTP_NOT_NEGOTIATED", ERR_LIB_SSL, 369}, + #endif + #ifdef SSL_R_VERSION_TOO_LOW + {"VERSION_TOO_LOW", ERR_LIB_SSL, SSL_R_VERSION_TOO_LOW}, + #else + {"VERSION_TOO_LOW", ERR_LIB_SSL, 396}, + #endif #ifdef SSL_R_WRITE_BIO_NOT_SET {"WRITE_BIO_NOT_SET", ERR_LIB_SSL, SSL_R_WRITE_BIO_NOT_SET}, #else {"WRITE_BIO_NOT_SET", ERR_LIB_SSL, 260}, #endif + #ifdef SSL_R_WRONG_CERTIFICATE_TYPE + {"WRONG_CERTIFICATE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_CERTIFICATE_TYPE}, + #else + {"WRONG_CERTIFICATE_TYPE", ERR_LIB_SSL, 383}, + #endif #ifdef SSL_R_WRONG_CIPHER_RETURNED {"WRONG_CIPHER_RETURNED", ERR_LIB_SSL, SSL_R_WRONG_CIPHER_RETURNED}, #else {"WRONG_CIPHER_RETURNED", ERR_LIB_SSL, 261}, #endif + #ifdef SSL_R_WRONG_CURVE + {"WRONG_CURVE", ERR_LIB_SSL, SSL_R_WRONG_CURVE}, + #else + {"WRONG_CURVE", ERR_LIB_SSL, 378}, + #endif #ifdef SSL_R_WRONG_MESSAGE_TYPE {"WRONG_MESSAGE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_MESSAGE_TYPE}, #else @@ -1499,6 +1754,11 @@ #else {"WRONG_SIGNATURE_SIZE", ERR_LIB_SSL, 265}, #endif + #ifdef SSL_R_WRONG_SIGNATURE_TYPE + {"WRONG_SIGNATURE_TYPE", ERR_LIB_SSL, SSL_R_WRONG_SIGNATURE_TYPE}, + #else + {"WRONG_SIGNATURE_TYPE", ERR_LIB_SSL, 370}, + #endif #ifdef SSL_R_WRONG_SSL_VERSION {"WRONG_SSL_VERSION", ERR_LIB_SSL, SSL_R_WRONG_SSL_VERSION}, #else @@ -1519,6 +1779,11 @@ #else {"X509_VERIFICATION_SETUP_PROBLEMS", ERR_LIB_SSL, 269}, #endif + #ifdef X509_R_AKID_MISMATCH + {"AKID_MISMATCH", ERR_LIB_X509, X509_R_AKID_MISMATCH}, + #else + {"AKID_MISMATCH", ERR_LIB_X509, 110}, + #endif #ifdef X509_R_BAD_X509_FILETYPE {"BAD_X509_FILETYPE", ERR_LIB_X509, X509_R_BAD_X509_FILETYPE}, #else @@ -1539,11 +1804,26 @@ #else {"CERT_ALREADY_IN_HASH_TABLE", ERR_LIB_X509, 101}, #endif + #ifdef X509_R_CRL_ALREADY_DELTA + {"CRL_ALREADY_DELTA", ERR_LIB_X509, X509_R_CRL_ALREADY_DELTA}, + #else + {"CRL_ALREADY_DELTA", ERR_LIB_X509, 127}, + #endif + #ifdef X509_R_CRL_VERIFY_FAILURE + {"CRL_VERIFY_FAILURE", ERR_LIB_X509, X509_R_CRL_VERIFY_FAILURE}, + #else + {"CRL_VERIFY_FAILURE", ERR_LIB_X509, 131}, + #endif #ifdef X509_R_ERR_ASN1_LIB {"ERR_ASN1_LIB", ERR_LIB_X509, X509_R_ERR_ASN1_LIB}, #else {"ERR_ASN1_LIB", ERR_LIB_X509, 102}, #endif + #ifdef X509_R_IDP_MISMATCH + {"IDP_MISMATCH", ERR_LIB_X509, X509_R_IDP_MISMATCH}, + #else + {"IDP_MISMATCH", ERR_LIB_X509, 128}, + #endif #ifdef X509_R_INVALID_DIRECTORY {"INVALID_DIRECTORY", ERR_LIB_X509, X509_R_INVALID_DIRECTORY}, #else @@ -1559,6 +1839,11 @@ #else {"INVALID_TRUST", ERR_LIB_X509, 123}, #endif + #ifdef X509_R_ISSUER_MISMATCH + {"ISSUER_MISMATCH", ERR_LIB_X509, X509_R_ISSUER_MISMATCH}, + #else + {"ISSUER_MISMATCH", ERR_LIB_X509, 129}, + #endif #ifdef X509_R_KEY_TYPE_MISMATCH {"KEY_TYPE_MISMATCH", ERR_LIB_X509, X509_R_KEY_TYPE_MISMATCH}, #else @@ -1584,11 +1869,21 @@ #else {"METHOD_NOT_SUPPORTED", ERR_LIB_X509, 124}, #endif + #ifdef X509_R_NEWER_CRL_NOT_NEWER + {"NEWER_CRL_NOT_NEWER", ERR_LIB_X509, X509_R_NEWER_CRL_NOT_NEWER}, + #else + {"NEWER_CRL_NOT_NEWER", ERR_LIB_X509, 132}, + #endif #ifdef X509_R_NO_CERT_SET_FOR_US_TO_VERIFY {"NO_CERT_SET_FOR_US_TO_VERIFY", ERR_LIB_X509, X509_R_NO_CERT_SET_FOR_US_TO_VERIFY}, #else {"NO_CERT_SET_FOR_US_TO_VERIFY", ERR_LIB_X509, 105}, #endif + #ifdef X509_R_NO_CRL_NUMBER + {"NO_CRL_NUMBER", ERR_LIB_X509, X509_R_NO_CRL_NUMBER}, + #else + {"NO_CRL_NUMBER", ERR_LIB_X509, 130}, + #endif #ifdef X509_R_PUBLIC_KEY_DECODE_ERROR {"PUBLIC_KEY_DECODE_ERROR", ERR_LIB_X509, X509_R_PUBLIC_KEY_DECODE_ERROR}, #else diff --git a/Tools/ssl/make_ssl_data.py b/Tools/ssl/make_ssl_data.py --- a/Tools/ssl/make_ssl_data.py +++ b/Tools/ssl/make_ssl_data.py @@ -5,8 +5,7 @@ `library` and `reason` mnemnonics to a more recent OpenSSL version. It takes two arguments: -- the path to the OpenSSL include files' directory - (e.g. openssl-1.0.1-beta3/include/openssl/) +- the path to the OpenSSL source tree (e.g. git checkout) - the path to the C file to be generated (probably Modules/_ssl_data.h) """ @@ -15,9 +14,10 @@ import os import re import sys +import _ssl -def parse_error_codes(h_file, prefix): +def parse_error_codes(h_file, prefix, libcode): pat = re.compile(r"#define\W+(%s([\w]+))\W+(\d+)\b" % re.escape(prefix)) codes = [] with open(h_file, "r", encoding="latin1") as f: @@ -26,7 +26,8 @@ if match: code, name, num = match.groups() num = int(num) - codes.append((code, name, num)) + # e.g. ("SSL_R_BAD_DATA", ("ERR_LIB_SSL", "BAD_DATA", 390)) + codes.append((code, (libcode, name, num))) return codes if __name__ == "__main__": @@ -34,12 +35,32 @@ outfile = sys.argv[2] use_stdout = outfile == '-' f = sys.stdout if use_stdout else open(outfile, "w") - error_libraries = ( - # (library code, mnemonic, error prefix, header file) - ('ERR_LIB_PEM', 'PEM', 'PEM_R_', 'pem.h'), - ('ERR_LIB_SSL', 'SSL', 'SSL_R_', 'ssl.h'), - ('ERR_LIB_X509', 'X509', 'X509_R_', 'x509.h'), - ) + error_libraries = { + # mnemonic -> (library code, error prefix, header file) + 'PEM': ('ERR_LIB_PEM', 'PEM_R_', 'crypto/pem/pem.h'), + 'SSL': ('ERR_LIB_SSL', 'SSL_R_', 'ssl/ssl.h'), + 'X509': ('ERR_LIB_X509', 'X509_R_', 'crypto/x509/x509.h'), + } + + # Read codes from libraries + new_codes = [] + for libcode, prefix, h_file in sorted(error_libraries.values()): + new_codes += parse_error_codes(os.path.join(openssl_inc, h_file), + prefix, libcode) + new_code_nums = set((libcode, num) + for (code, (libcode, name, num)) in new_codes) + + # Merge with existing codes (in case some old codes disappeared). + codes = {} + for errname, (libnum, errnum) in _ssl.err_names_to_codes.items(): + lib = error_libraries[_ssl.lib_codes_to_names[libnum]] + libcode = lib[0] # e.g. ERR_LIB_PEM + errcode = lib[1] + errname # e.g. SSL_R_BAD_SSL_SESSION_ID_LENGTH + # Only keep it if the numeric codes weren't reused + if (libcode, errnum) not in new_code_nums: + codes[errcode] = libcode, errname, errnum + codes.update(dict(new_codes)) + def w(l): f.write(l + "\n") w("/* File generated by Tools/ssl/make_ssl_data.py */") @@ -47,21 +68,19 @@ w("") w("static struct py_ssl_library_code library_codes[] = {") - for libcode, mnemo, _, _ in error_libraries: + for mnemo, (libcode, _, _) in sorted(error_libraries.items()): w(' {"%s", %s},' % (mnemo, libcode)) w(' { NULL }') w('};') w("") w("static struct py_ssl_error_code error_codes[] = {") - for libcode, _, prefix, h_file in error_libraries: - codes = parse_error_codes(os.path.join(openssl_inc, h_file), prefix) - for code, name, num in sorted(codes): - w(' #ifdef %s' % (code)) - w(' {"%s", %s, %s},' % (name, libcode, code)) - w(' #else') - w(' {"%s", %s, %d},' % (name, libcode, num)) - w(' #endif') + for errcode, (libcode, name, num) in sorted(codes.items()): + w(' #ifdef %s' % (errcode)) + w(' {"%s", %s, %s},' % (name, libcode, errcode)) + w(' #else') + w(' {"%s", %s, %d},' % (name, libcode, num)) + w(' #endif') w(' { NULL }') w('};') if not use_stdout: diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -696,7 +696,7 @@ exts.append( Extension('audioop', ['audioop.c']) ) # Disabled on 64-bit platforms - if sys.maxint != 9223372036854775807L: + if sys.maxsize != 9223372036854775807L: # Operations on images exts.append( Extension('imageop', ['imageop.c']) ) else: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:39:44 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:39:44 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323094=3A_Fixed_readline_with_frames_in_Python_i?= =?utf-8?q?mplementation_of_pickle=2E?= Message-ID: <20150126083820.87107.27009@psf.io> https://hg.python.org/cpython/rev/c347c21e5afa changeset: 94297:c347c21e5afa parent: 94294:1db1cd711104 parent: 94296:d5e13b74d377 user: Serhiy Storchaka date: Mon Jan 26 10:37:44 2015 +0200 summary: Issue #23094: Fixed readline with frames in Python implementation of pickle. files: Lib/pickle.py | 2 +- Lib/test/pickletester.py | 8 ++++++++ Misc/NEWS | 2 ++ 3 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -242,7 +242,7 @@ if not data: self.current_frame = None return self.file_readline() - if data[-1] != b'\n': + if data[-1] != b'\n'[0]: raise UnpicklingError( "pickle exhausted before end of frame") return data diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -1584,6 +1584,14 @@ count_opcode(pickle.FRAME, pickled)) self.assertEqual(obj, self.loads(some_frames_pickle)) + def test_frame_readline(self): + pickled = b'\x80\x04\x95\x05\x00\x00\x00\x00\x00\x00\x00I42\n.' + # 0: \x80 PROTO 4 + # 2: \x95 FRAME 5 + # 11: I INT 42 + # 15: . STOP + self.assertEqual(self.loads(pickled), 42) + def test_nested_names(self): global Nested class Nested: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -218,6 +218,8 @@ Library ------- +- Issue #23094: Fixed readline with frames in Python implementation of pickle. + - Issue #23268: Fixed bugs in the comparison of ipaddress classes. - Issue #21408: Removed incorrect implementations of __ne__() which didn't -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 09:39:44 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 08:39:44 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDk0?= =?utf-8?q?=3A_Fixed_readline_with_frames_in_Python_implementation_of_pick?= =?utf-8?b?bGUu?= Message-ID: <20150126083820.118224.7405@psf.io> https://hg.python.org/cpython/rev/d5e13b74d377 changeset: 94296:d5e13b74d377 branch: 3.4 parent: 94293:04fa56628830 user: Serhiy Storchaka date: Mon Jan 26 10:37:01 2015 +0200 summary: Issue #23094: Fixed readline with frames in Python implementation of pickle. files: Lib/pickle.py | 2 +- Lib/test/pickletester.py | 8 ++++++++ Misc/NEWS | 2 ++ 3 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -242,7 +242,7 @@ if not data: self.current_frame = None return self.file_readline() - if data[-1] != b'\n': + if data[-1] != b'\n'[0]: raise UnpicklingError( "pickle exhausted before end of frame") return data diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -1538,6 +1538,14 @@ count_opcode(pickle.FRAME, pickled)) self.assertEqual(obj, self.loads(some_frames_pickle)) + def test_frame_readline(self): + pickled = b'\x80\x04\x95\x05\x00\x00\x00\x00\x00\x00\x00I42\n.' + # 0: \x80 PROTO 4 + # 2: \x95 FRAME 5 + # 11: I INT 42 + # 15: . STOP + self.assertEqual(self.loads(pickled), 42) + def test_nested_names(self): global Nested class Nested: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -50,6 +50,8 @@ Library ------- +- Issue #23094: Fixed readline with frames in Python implementation of pickle. + - Issue #23268: Fixed bugs in the comparison of ipaddress classes. - Issue #21408: Removed incorrect implementations of __ne__() which didn't -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:07:31 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 10:07:31 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQ2xv?= =?utf-8?q?se_transports_on_error?= Message-ID: <20150126100729.75796.96847@psf.io> https://hg.python.org/cpython/rev/a9e3de181df4 changeset: 94298:a9e3de181df4 branch: 3.4 parent: 94296:d5e13b74d377 user: Victor Stinner date: Mon Jan 26 11:02:18 2015 +0100 summary: asyncio: Close transports on error Fix create_datagram_endpoint(), connect_read_pipe() and connect_write_pipe(): close the transport if the task is cancelled or on error. files: Lib/asyncio/base_events.py | 24 +++++++++++++++++++++--- 1 files changed, 21 insertions(+), 3 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -723,7 +723,13 @@ logger.debug("Datagram endpoint remote_addr=%r created: " "(%r, %r)", remote_addr, transport, protocol) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + return transport, protocol @coroutine @@ -815,7 +821,13 @@ protocol = protocol_factory() waiter = futures.Future(loop=self) transport = self._make_read_pipe_transport(pipe, protocol, waiter) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + if self._debug: logger.debug('Read pipe %r connected: (%r, %r)', pipe.fileno(), transport, protocol) @@ -826,7 +838,13 @@ protocol = protocol_factory() waiter = futures.Future(loop=self) transport = self._make_write_pipe_transport(pipe, protocol, waiter) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + if self._debug: logger.debug('Write pipe %r connected: (%r, %r)', pipe.fileno(), transport, protocol) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:07:31 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 10:07:31 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjA4?= =?utf-8?q?=2C_asyncio=3A_Add_BaseEventLoop=2E=5Fcurrent=5Fhandle?= Message-ID: <20150126100730.55124.95344@psf.io> https://hg.python.org/cpython/rev/54d74f954bf9 changeset: 94300:54d74f954bf9 branch: 3.4 user: Victor Stinner date: Mon Jan 26 11:05:12 2015 +0100 summary: Issue #23208, asyncio: Add BaseEventLoop._current_handle In debug mode, BaseEventLoop._run_once() now sets the BaseEventLoop._current_handle attribute to the handle currently executed. In release mode or when no handle is executed, the attribute is None. BaseEventLoop.default_exception_handler() displays the traceback of the current handle if available. files: Lib/asyncio/base_events.py | 25 +++++++++++++++++++------ 1 files changed, 19 insertions(+), 6 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -179,6 +179,7 @@ # In debug mode, if the execution of a callback or a step of a task # exceed this duration in seconds, the slow callback/task is logged. self.slow_callback_duration = 0.1 + self._current_handle = None def __repr__(self): return ('<%s running=%s closed=%s debug=%s>' @@ -955,6 +956,10 @@ else: exc_info = False + if (self._current_handle is not None + and self._current_handle._source_traceback): + context['handle_traceback'] = self._current_handle._source_traceback + log_lines = [message] for key in sorted(context): if key in {'message', 'exception'}: @@ -964,6 +969,10 @@ tb = ''.join(traceback.format_list(value)) value = 'Object created at (most recent call last):\n' value += tb.rstrip() + elif key == 'handle_traceback': + tb = ''.join(traceback.format_list(value)) + value = 'Handle created at (most recent call last):\n' + value += tb.rstrip() else: value = repr(value) log_lines.append('{}: {}'.format(key, value)) @@ -1121,12 +1130,16 @@ if handle._cancelled: continue if self._debug: - t0 = self.time() - handle._run() - dt = self.time() - t0 - if dt >= self.slow_callback_duration: - logger.warning('Executing %s took %.3f seconds', - _format_handle(handle), dt) + try: + self._current_handle = handle + t0 = self.time() + handle._run() + dt = self.time() - t0 + if dt >= self.slow_callback_duration: + logger.warning('Executing %s took %.3f seconds', + _format_handle(handle), dt) + finally: + self._current_handle = None else: handle._run() handle = None # Needed to break cycles when an exception occurs. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:07:31 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 10:07:31 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjkz?= =?utf-8?q?=2C_asyncio=3A_Cleanup_IocpProactor=2Eclose=28=29?= Message-ID: <20150126100730.84283.1599@psf.io> https://hg.python.org/cpython/rev/b6ab8fe16d16 changeset: 94299:b6ab8fe16d16 branch: 3.4 user: Victor Stinner date: Mon Jan 26 11:02:59 2015 +0100 summary: Issue #23293, asyncio: Cleanup IocpProactor.close() The special case for connect_pipe() is not more needed. connect_pipe() doesn't use overlapped operations anymore. files: Lib/asyncio/windows_events.py | 7 +------ 1 files changed, 1 insertions(+), 6 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -694,12 +694,7 @@ def close(self): # Cancel remaining registered operations. for address, (fut, ov, obj, callback) in list(self._cache.items()): - if obj is None: - # The operation was started with connect_pipe() which - # queues a task to Windows' thread pool. This cannot - # be cancelled, so just forget it. - del self._cache[address] - elif fut.cancelled(): + if fut.cancelled(): # Nothing to do with cancelled futures pass elif isinstance(fut, _WaitCancelFuture): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:07:31 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 10:07:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150126100730.87119.91316@psf.io> https://hg.python.org/cpython/rev/b31dae6f3364 changeset: 94301:b31dae6f3364 parent: 94297:c347c21e5afa parent: 94300:54d74f954bf9 user: Victor Stinner date: Mon Jan 26 11:05:34 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 49 ++++++++++++++++++---- Lib/asyncio/windows_events.py | 7 +-- 2 files changed, 41 insertions(+), 15 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -179,6 +179,7 @@ # In debug mode, if the execution of a callback or a step of a task # exceed this duration in seconds, the slow callback/task is logged. self.slow_callback_duration = 0.1 + self._current_handle = None def __repr__(self): return ('<%s running=%s closed=%s debug=%s>' @@ -723,7 +724,13 @@ logger.debug("Datagram endpoint remote_addr=%r created: " "(%r, %r)", remote_addr, transport, protocol) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + return transport, protocol @coroutine @@ -815,7 +822,13 @@ protocol = protocol_factory() waiter = futures.Future(loop=self) transport = self._make_read_pipe_transport(pipe, protocol, waiter) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + if self._debug: logger.debug('Read pipe %r connected: (%r, %r)', pipe.fileno(), transport, protocol) @@ -826,7 +839,13 @@ protocol = protocol_factory() waiter = futures.Future(loop=self) transport = self._make_write_pipe_transport(pipe, protocol, waiter) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + if self._debug: logger.debug('Write pipe %r connected: (%r, %r)', pipe.fileno(), transport, protocol) @@ -937,6 +956,10 @@ else: exc_info = False + if (self._current_handle is not None + and self._current_handle._source_traceback): + context['handle_traceback'] = self._current_handle._source_traceback + log_lines = [message] for key in sorted(context): if key in {'message', 'exception'}: @@ -946,6 +969,10 @@ tb = ''.join(traceback.format_list(value)) value = 'Object created at (most recent call last):\n' value += tb.rstrip() + elif key == 'handle_traceback': + tb = ''.join(traceback.format_list(value)) + value = 'Handle created at (most recent call last):\n' + value += tb.rstrip() else: value = repr(value) log_lines.append('{}: {}'.format(key, value)) @@ -1103,12 +1130,16 @@ if handle._cancelled: continue if self._debug: - t0 = self.time() - handle._run() - dt = self.time() - t0 - if dt >= self.slow_callback_duration: - logger.warning('Executing %s took %.3f seconds', - _format_handle(handle), dt) + try: + self._current_handle = handle + t0 = self.time() + handle._run() + dt = self.time() - t0 + if dt >= self.slow_callback_duration: + logger.warning('Executing %s took %.3f seconds', + _format_handle(handle), dt) + finally: + self._current_handle = None else: handle._run() handle = None # Needed to break cycles when an exception occurs. diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -694,12 +694,7 @@ def close(self): # Cancel remaining registered operations. for address, (fut, ov, obj, callback) in list(self._cache.items()): - if obj is None: - # The operation was started with connect_pipe() which - # queues a task to Windows' thread pool. This cannot - # be cancelled, so just forget it. - del self._cache[address] - elif fut.cancelled(): + if fut.cancelled(): # Nothing to do with cancelled futures pass elif isinstance(fut, _WaitCancelFuture): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:15:31 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 10:15:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <20150126101525.118206.94589@psf.io> https://hg.python.org/cpython/rev/3c11b3184b83 changeset: 94308:3c11b3184b83 parent: 94307:a6c881abba49 parent: 94306:146fe233ecc0 user: Serhiy Storchaka date: Mon Jan 26 12:14:37 2015 +0200 summary: Null merge files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:15:31 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 10:15:31 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE4NTE4?= =?utf-8?q?=3A_timeit_now_rejects_statements_which_can=27t_be_compiled_out?= =?utf-8?q?side?= Message-ID: <20150126101524.55102.40496@psf.io> https://hg.python.org/cpython/rev/a5769fa55791 changeset: 94304:a5769fa55791 branch: 3.4 parent: 94296:d5e13b74d377 user: Serhiy Storchaka date: Mon Jan 26 12:09:17 2015 +0200 summary: Issue #18518: timeit now rejects statements which can't be compiled outside a function or a loop (e.g. "return" or "break"). files: Doc/library/timeit.rst | 6 ------ Lib/test/test_timeit.py | 12 ++++++++++++ Lib/timeit.py | 6 ++++++ Misc/NEWS | 3 +++ 4 files changed, 21 insertions(+), 6 deletions(-) diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -64,12 +64,6 @@ Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.timeit` method with *number* executions. - .. note:: - - Because :meth:`.timeit` is executing *stmt*, placing a return statement - in *stmt* will prevent :meth:`.timeit` from returning execution time. - It will instead return the data specified by your return statement. - .. function:: repeat(stmt='pass', setup='pass', timer=, repeat=3, number=1000000) diff --git a/Lib/test/test_timeit.py b/Lib/test/test_timeit.py --- a/Lib/test/test_timeit.py +++ b/Lib/test/test_timeit.py @@ -73,9 +73,21 @@ def test_timer_invalid_stmt(self): self.assertRaises(ValueError, timeit.Timer, stmt=None) + self.assertRaises(SyntaxError, timeit.Timer, stmt='return') + self.assertRaises(SyntaxError, timeit.Timer, stmt='yield') + self.assertRaises(SyntaxError, timeit.Timer, stmt='yield from ()') + self.assertRaises(SyntaxError, timeit.Timer, stmt='break') + self.assertRaises(SyntaxError, timeit.Timer, stmt='continue') + self.assertRaises(SyntaxError, timeit.Timer, stmt='from timeit import *') def test_timer_invalid_setup(self): self.assertRaises(ValueError, timeit.Timer, setup=None) + self.assertRaises(SyntaxError, timeit.Timer, setup='return') + self.assertRaises(SyntaxError, timeit.Timer, setup='yield') + self.assertRaises(SyntaxError, timeit.Timer, setup='yield from ()') + self.assertRaises(SyntaxError, timeit.Timer, setup='break') + self.assertRaises(SyntaxError, timeit.Timer, setup='continue') + self.assertRaises(SyntaxError, timeit.Timer, setup='from timeit import *') fake_setup = "import timeit; timeit._fake_timer.setup()" fake_stmt = "import timeit; timeit._fake_timer.inc()" diff --git a/Lib/timeit.py b/Lib/timeit.py --- a/Lib/timeit.py +++ b/Lib/timeit.py @@ -109,6 +109,12 @@ self.timer = timer ns = {} if isinstance(stmt, str): + # Check that the code can be compiled outside a function + if isinstance(setup, str): + compile(setup, dummy_src_name, "exec") + compile(setup + '\n' + stmt, dummy_src_name, "exec") + else: + compile(stmt, dummy_src_name, "exec") stmt = reindent(stmt, 8) if isinstance(setup, str): setup = reindent(setup, 4) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -50,6 +50,9 @@ Library ------- +- Issue #18518: timeit now rejects statements which can't be compiled outside + a function or a loop (e.g. "return" or "break"). + - Issue #23094: Fixed readline with frames in Python implementation of pickle. - Issue #23268: Fixed bugs in the comparison of ipaddress classes. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:15:31 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 10:15:31 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy40IC0+IDMuNCk6?= =?utf-8?q?_Merge_heads?= Message-ID: <20150126101524.84275.7304@psf.io> https://hg.python.org/cpython/rev/146fe233ecc0 changeset: 94306:146fe233ecc0 branch: 3.4 parent: 94304:a5769fa55791 parent: 94300:54d74f954bf9 user: Serhiy Storchaka date: Mon Jan 26 12:12:31 2015 +0200 summary: Merge heads files: Lib/asyncio/base_events.py | 49 ++++++++++++++++++---- Lib/asyncio/windows_events.py | 7 +-- 2 files changed, 41 insertions(+), 15 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -179,6 +179,7 @@ # In debug mode, if the execution of a callback or a step of a task # exceed this duration in seconds, the slow callback/task is logged. self.slow_callback_duration = 0.1 + self._current_handle = None def __repr__(self): return ('<%s running=%s closed=%s debug=%s>' @@ -723,7 +724,13 @@ logger.debug("Datagram endpoint remote_addr=%r created: " "(%r, %r)", remote_addr, transport, protocol) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + return transport, protocol @coroutine @@ -815,7 +822,13 @@ protocol = protocol_factory() waiter = futures.Future(loop=self) transport = self._make_read_pipe_transport(pipe, protocol, waiter) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + if self._debug: logger.debug('Read pipe %r connected: (%r, %r)', pipe.fileno(), transport, protocol) @@ -826,7 +839,13 @@ protocol = protocol_factory() waiter = futures.Future(loop=self) transport = self._make_write_pipe_transport(pipe, protocol, waiter) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + if self._debug: logger.debug('Write pipe %r connected: (%r, %r)', pipe.fileno(), transport, protocol) @@ -937,6 +956,10 @@ else: exc_info = False + if (self._current_handle is not None + and self._current_handle._source_traceback): + context['handle_traceback'] = self._current_handle._source_traceback + log_lines = [message] for key in sorted(context): if key in {'message', 'exception'}: @@ -946,6 +969,10 @@ tb = ''.join(traceback.format_list(value)) value = 'Object created at (most recent call last):\n' value += tb.rstrip() + elif key == 'handle_traceback': + tb = ''.join(traceback.format_list(value)) + value = 'Handle created at (most recent call last):\n' + value += tb.rstrip() else: value = repr(value) log_lines.append('{}: {}'.format(key, value)) @@ -1103,12 +1130,16 @@ if handle._cancelled: continue if self._debug: - t0 = self.time() - handle._run() - dt = self.time() - t0 - if dt >= self.slow_callback_duration: - logger.warning('Executing %s took %.3f seconds', - _format_handle(handle), dt) + try: + self._current_handle = handle + t0 = self.time() + handle._run() + dt = self.time() - t0 + if dt >= self.slow_callback_duration: + logger.warning('Executing %s took %.3f seconds', + _format_handle(handle), dt) + finally: + self._current_handle = None else: handle._run() handle = None # Needed to break cycles when an exception occurs. diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -694,12 +694,7 @@ def close(self): # Cancel remaining registered operations. for address, (fut, ov, obj, callback) in list(self._cache.items()): - if obj is None: - # The operation was started with connect_pipe() which - # queues a task to Windows' thread pool. This cannot - # be cancelled, so just forget it. - del self._cache[address] - elif fut.cancelled(): + if fut.cancelled(): # Nothing to do with cancelled futures pass elif isinstance(fut, _WaitCancelFuture): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:15:32 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 10:15:32 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4NTE4?= =?utf-8?q?=3A_timeit_now_rejects_statements_which_can=27t_be_compiled_out?= =?utf-8?q?side?= Message-ID: <20150126101524.75798.78871@psf.io> https://hg.python.org/cpython/rev/e8db1cbe416b changeset: 94303:e8db1cbe416b branch: 2.7 user: Serhiy Storchaka date: Mon Jan 26 12:08:37 2015 +0200 summary: Issue #18518: timeit now rejects statements which can't be compiled outside a function or a loop (e.g. "return" or "break"). files: Lib/test/test_timeit.py | 8 ++++++++ Lib/timeit.py | 6 ++++++ Misc/NEWS | 3 +++ 3 files changed, 17 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_timeit.py b/Lib/test/test_timeit.py --- a/Lib/test/test_timeit.py +++ b/Lib/test/test_timeit.py @@ -73,9 +73,17 @@ def test_timer_invalid_stmt(self): self.assertRaises(ValueError, timeit.Timer, stmt=None) + self.assertRaises(SyntaxError, timeit.Timer, stmt='return') + self.assertRaises(SyntaxError, timeit.Timer, stmt='yield') + self.assertRaises(SyntaxError, timeit.Timer, stmt='break') + self.assertRaises(SyntaxError, timeit.Timer, stmt='continue') def test_timer_invalid_setup(self): self.assertRaises(ValueError, timeit.Timer, setup=None) + self.assertRaises(SyntaxError, timeit.Timer, setup='return') + self.assertRaises(SyntaxError, timeit.Timer, setup='yield') + self.assertRaises(SyntaxError, timeit.Timer, setup='break') + self.assertRaises(SyntaxError, timeit.Timer, setup='continue') fake_setup = "import timeit; timeit._fake_timer.setup()" fake_stmt = "import timeit; timeit._fake_timer.inc()" diff --git a/Lib/timeit.py b/Lib/timeit.py --- a/Lib/timeit.py +++ b/Lib/timeit.py @@ -123,6 +123,12 @@ self.timer = timer ns = {} if isinstance(stmt, basestring): + # Check that the code can be compiled outside a function + if isinstance(setup, basestring): + compile(setup, dummy_src_name, "exec") + compile(setup + '\n' + stmt, dummy_src_name, "exec") + else: + compile(stmt, dummy_src_name, "exec") stmt = reindent(stmt, 8) if isinstance(setup, basestring): setup = reindent(setup, 4) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,9 @@ Library ------- +- Issue #18518: timeit now rejects statements which can't be compiled outside + a function or a loop (e.g. "return" or "break"). + - Issue #19996: Make :mod:`httplib` ignore headers with no name rather than assuming the body has started. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:15:32 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 10:15:32 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2318518=3A_timeit_now_rejects_statements_which_ca?= =?utf-8?q?n=27t_be_compiled_outside?= Message-ID: <20150126101524.84261.8795@psf.io> https://hg.python.org/cpython/rev/b0a686260b5d changeset: 94305:b0a686260b5d parent: 94297:c347c21e5afa parent: 94304:a5769fa55791 user: Serhiy Storchaka date: Mon Jan 26 12:09:59 2015 +0200 summary: Issue #18518: timeit now rejects statements which can't be compiled outside a function or a loop (e.g. "return" or "break"). files: Doc/library/timeit.rst | 6 ------ Lib/test/test_timeit.py | 12 ++++++++++++ Lib/timeit.py | 6 ++++++ Misc/NEWS | 3 +++ 4 files changed, 21 insertions(+), 6 deletions(-) diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -69,12 +69,6 @@ .. versionchanged:: 3.5 The optional *globals* parameter was added. - .. note:: - - Because :meth:`.timeit` is executing *stmt*, placing a return statement - in *stmt* will prevent :meth:`.timeit` from returning execution time. - It will instead return the data specified by your return statement. - .. function:: repeat(stmt='pass', setup='pass', timer=, repeat=3, number=1000000, globals=None) diff --git a/Lib/test/test_timeit.py b/Lib/test/test_timeit.py --- a/Lib/test/test_timeit.py +++ b/Lib/test/test_timeit.py @@ -73,9 +73,21 @@ def test_timer_invalid_stmt(self): self.assertRaises(ValueError, timeit.Timer, stmt=None) + self.assertRaises(SyntaxError, timeit.Timer, stmt='return') + self.assertRaises(SyntaxError, timeit.Timer, stmt='yield') + self.assertRaises(SyntaxError, timeit.Timer, stmt='yield from ()') + self.assertRaises(SyntaxError, timeit.Timer, stmt='break') + self.assertRaises(SyntaxError, timeit.Timer, stmt='continue') + self.assertRaises(SyntaxError, timeit.Timer, stmt='from timeit import *') def test_timer_invalid_setup(self): self.assertRaises(ValueError, timeit.Timer, setup=None) + self.assertRaises(SyntaxError, timeit.Timer, setup='return') + self.assertRaises(SyntaxError, timeit.Timer, setup='yield') + self.assertRaises(SyntaxError, timeit.Timer, setup='yield from ()') + self.assertRaises(SyntaxError, timeit.Timer, setup='break') + self.assertRaises(SyntaxError, timeit.Timer, setup='continue') + self.assertRaises(SyntaxError, timeit.Timer, setup='from timeit import *') fake_setup = "import timeit; timeit._fake_timer.setup()" fake_stmt = "import timeit; timeit._fake_timer.inc()" diff --git a/Lib/timeit.py b/Lib/timeit.py --- a/Lib/timeit.py +++ b/Lib/timeit.py @@ -115,6 +115,12 @@ local_ns = {} global_ns = _globals() if globals is None else globals if isinstance(stmt, str): + # Check that the code can be compiled outside a function + if isinstance(setup, str): + compile(setup, dummy_src_name, "exec") + compile(setup + '\n' + stmt, dummy_src_name, "exec") + else: + compile(stmt, dummy_src_name, "exec") stmt = reindent(stmt, 8) if isinstance(setup, str): setup = reindent(setup, 4) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -218,6 +218,9 @@ Library ------- +- Issue #18518: timeit now rejects statements which can't be compiled outside + a function or a loop (e.g. "return" or "break"). + - Issue #23094: Fixed readline with frames in Python implementation of pickle. - Issue #23268: Fixed bugs in the comparison of ipaddress classes. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:15:32 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 10:15:32 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <20150126101525.75776.96626@psf.io> https://hg.python.org/cpython/rev/a6c881abba49 changeset: 94307:a6c881abba49 parent: 94305:b0a686260b5d parent: 94301:b31dae6f3364 user: Serhiy Storchaka date: Mon Jan 26 12:13:09 2015 +0200 summary: Merge heads files: Lib/asyncio/base_events.py | 49 ++++++++++++++++++---- Lib/asyncio/windows_events.py | 7 +-- 2 files changed, 41 insertions(+), 15 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -179,6 +179,7 @@ # In debug mode, if the execution of a callback or a step of a task # exceed this duration in seconds, the slow callback/task is logged. self.slow_callback_duration = 0.1 + self._current_handle = None def __repr__(self): return ('<%s running=%s closed=%s debug=%s>' @@ -723,7 +724,13 @@ logger.debug("Datagram endpoint remote_addr=%r created: " "(%r, %r)", remote_addr, transport, protocol) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + return transport, protocol @coroutine @@ -815,7 +822,13 @@ protocol = protocol_factory() waiter = futures.Future(loop=self) transport = self._make_read_pipe_transport(pipe, protocol, waiter) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + if self._debug: logger.debug('Read pipe %r connected: (%r, %r)', pipe.fileno(), transport, protocol) @@ -826,7 +839,13 @@ protocol = protocol_factory() waiter = futures.Future(loop=self) transport = self._make_write_pipe_transport(pipe, protocol, waiter) - yield from waiter + + try: + yield from waiter + except: + transport.close() + raise + if self._debug: logger.debug('Write pipe %r connected: (%r, %r)', pipe.fileno(), transport, protocol) @@ -937,6 +956,10 @@ else: exc_info = False + if (self._current_handle is not None + and self._current_handle._source_traceback): + context['handle_traceback'] = self._current_handle._source_traceback + log_lines = [message] for key in sorted(context): if key in {'message', 'exception'}: @@ -946,6 +969,10 @@ tb = ''.join(traceback.format_list(value)) value = 'Object created at (most recent call last):\n' value += tb.rstrip() + elif key == 'handle_traceback': + tb = ''.join(traceback.format_list(value)) + value = 'Handle created at (most recent call last):\n' + value += tb.rstrip() else: value = repr(value) log_lines.append('{}: {}'.format(key, value)) @@ -1103,12 +1130,16 @@ if handle._cancelled: continue if self._debug: - t0 = self.time() - handle._run() - dt = self.time() - t0 - if dt >= self.slow_callback_duration: - logger.warning('Executing %s took %.3f seconds', - _format_handle(handle), dt) + try: + self._current_handle = handle + t0 = self.time() + handle._run() + dt = self.time() - t0 + if dt >= self.slow_callback_duration: + logger.warning('Executing %s took %.3f seconds', + _format_handle(handle), dt) + finally: + self._current_handle = None else: handle._run() handle = None # Needed to break cycles when an exception occurs. diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -694,12 +694,7 @@ def close(self): # Cancel remaining registered operations. for address, (fut, ov, obj, callback) in list(self._cache.items()): - if obj is None: - # The operation was started with connect_pipe() which - # queues a task to Windows' thread pool. This cannot - # be cancelled, so just forget it. - del self._cache[address] - elif fut.cancelled(): + if fut.cancelled(): # Nothing to do with cancelled futures pass elif isinstance(fut, _WaitCancelFuture): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:15:32 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 10:15:32 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzExNTc4?= =?utf-8?q?=3A_Backported_test_for_the_timeit_module=2E?= Message-ID: <20150126101523.84255.64545@psf.io> https://hg.python.org/cpython/rev/7e7c825f75ad changeset: 94302:7e7c825f75ad branch: 2.7 parent: 94295:36ca5e765704 user: Serhiy Storchaka date: Mon Jan 26 11:54:32 2015 +0200 summary: Issue #11578: Backported test for the timeit module. files: Lib/test/test_sundry.py | 1 - Lib/test/test_timeit.py | 304 ++++++++++++++++++++++++++++ Lib/timeit.py | 10 +- Misc/NEWS | 2 + 4 files changed, 314 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_sundry.py b/Lib/test/test_sundry.py --- a/Lib/test/test_sundry.py +++ b/Lib/test/test_sundry.py @@ -71,7 +71,6 @@ import sunaudio import symbol import tabnanny - import timeit import toaiff import token try: diff --git a/Lib/test/test_timeit.py b/Lib/test/test_timeit.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_timeit.py @@ -0,0 +1,304 @@ +import timeit +import unittest +import sys +from StringIO import StringIO +import time +from textwrap import dedent + +from test.support import run_unittest +from test.support import captured_stdout +from test.support import captured_stderr + +# timeit's default number of iterations. +DEFAULT_NUMBER = 1000000 + +# timeit's default number of repetitions. +DEFAULT_REPEAT = 3 + +# XXX: some tests are commented out that would improve the coverage but take a +# long time to run because they test the default number of loops, which is +# large. The tests could be enabled if there was a way to override the default +# number of loops during testing, but this would require changing the signature +# of some functions that use the default as a default argument. + +class FakeTimer: + BASE_TIME = 42.0 + def __init__(self, seconds_per_increment=1.0): + self.count = 0 + self.setup_calls = 0 + self.seconds_per_increment=seconds_per_increment + timeit._fake_timer = self + + def __call__(self): + return self.BASE_TIME + self.count * self.seconds_per_increment + + def inc(self): + self.count += 1 + + def setup(self): + self.setup_calls += 1 + + def wrap_timer(self, timer): + """Records 'timer' and returns self as callable timer.""" + self.saved_timer = timer + return self + +class TestTimeit(unittest.TestCase): + + def tearDown(self): + try: + del timeit._fake_timer + except AttributeError: + pass + + def test_reindent_empty(self): + self.assertEqual(timeit.reindent("", 0), "") + self.assertEqual(timeit.reindent("", 4), "") + + def test_reindent_single(self): + self.assertEqual(timeit.reindent("pass", 0), "pass") + self.assertEqual(timeit.reindent("pass", 4), "pass") + + def test_reindent_multi_empty(self): + self.assertEqual(timeit.reindent("\n\n", 0), "\n\n") + self.assertEqual(timeit.reindent("\n\n", 4), "\n \n ") + + def test_reindent_multi(self): + self.assertEqual(timeit.reindent( + "print()\npass\nbreak", 0), + "print()\npass\nbreak") + self.assertEqual(timeit.reindent( + "print()\npass\nbreak", 4), + "print()\n pass\n break") + + def test_timer_invalid_stmt(self): + self.assertRaises(ValueError, timeit.Timer, stmt=None) + + def test_timer_invalid_setup(self): + self.assertRaises(ValueError, timeit.Timer, setup=None) + + fake_setup = "import timeit; timeit._fake_timer.setup()" + fake_stmt = "import timeit; timeit._fake_timer.inc()" + + def fake_callable_setup(self): + self.fake_timer.setup() + + def fake_callable_stmt(self): + self.fake_timer.inc() + + def timeit(self, stmt, setup, number=None): + self.fake_timer = FakeTimer() + t = timeit.Timer(stmt=stmt, setup=setup, timer=self.fake_timer) + kwargs = {} + if number is None: + number = DEFAULT_NUMBER + else: + kwargs['number'] = number + delta_time = t.timeit(**kwargs) + self.assertEqual(self.fake_timer.setup_calls, 1) + self.assertEqual(self.fake_timer.count, number) + self.assertEqual(delta_time, number) + + # Takes too long to run in debug build. + #def test_timeit_default_iters(self): + # self.timeit(self.fake_stmt, self.fake_setup) + + def test_timeit_zero_iters(self): + self.timeit(self.fake_stmt, self.fake_setup, number=0) + + def test_timeit_few_iters(self): + self.timeit(self.fake_stmt, self.fake_setup, number=3) + + def test_timeit_callable_stmt(self): + self.timeit(self.fake_callable_stmt, self.fake_setup, number=3) + + def test_timeit_callable_stmt_and_setup(self): + self.timeit(self.fake_callable_stmt, + self.fake_callable_setup, number=3) + + # Takes too long to run in debug build. + #def test_timeit_function(self): + # delta_time = timeit.timeit(self.fake_stmt, self.fake_setup, + # timer=FakeTimer()) + # self.assertEqual(delta_time, DEFAULT_NUMBER) + + def test_timeit_function_zero_iters(self): + delta_time = timeit.timeit(self.fake_stmt, self.fake_setup, number=0, + timer=FakeTimer()) + self.assertEqual(delta_time, 0) + + def repeat(self, stmt, setup, repeat=None, number=None): + self.fake_timer = FakeTimer() + t = timeit.Timer(stmt=stmt, setup=setup, timer=self.fake_timer) + kwargs = {} + if repeat is None: + repeat = DEFAULT_REPEAT + else: + kwargs['repeat'] = repeat + if number is None: + number = DEFAULT_NUMBER + else: + kwargs['number'] = number + delta_times = t.repeat(**kwargs) + self.assertEqual(self.fake_timer.setup_calls, repeat) + self.assertEqual(self.fake_timer.count, repeat * number) + self.assertEqual(delta_times, repeat * [float(number)]) + + # Takes too long to run in debug build. + #def test_repeat_default(self): + # self.repeat(self.fake_stmt, self.fake_setup) + + def test_repeat_zero_reps(self): + self.repeat(self.fake_stmt, self.fake_setup, repeat=0) + + def test_repeat_zero_iters(self): + self.repeat(self.fake_stmt, self.fake_setup, number=0) + + def test_repeat_few_reps_and_iters(self): + self.repeat(self.fake_stmt, self.fake_setup, repeat=3, number=5) + + def test_repeat_callable_stmt(self): + self.repeat(self.fake_callable_stmt, self.fake_setup, + repeat=3, number=5) + + def test_repeat_callable_stmt_and_setup(self): + self.repeat(self.fake_callable_stmt, self.fake_callable_setup, + repeat=3, number=5) + + # Takes too long to run in debug build. + #def test_repeat_function(self): + # delta_times = timeit.repeat(self.fake_stmt, self.fake_setup, + # timer=FakeTimer()) + # self.assertEqual(delta_times, DEFAULT_REPEAT * [float(DEFAULT_NUMBER)]) + + def test_repeat_function_zero_reps(self): + delta_times = timeit.repeat(self.fake_stmt, self.fake_setup, repeat=0, + timer=FakeTimer()) + self.assertEqual(delta_times, []) + + def test_repeat_function_zero_iters(self): + delta_times = timeit.repeat(self.fake_stmt, self.fake_setup, number=0, + timer=FakeTimer()) + self.assertEqual(delta_times, DEFAULT_REPEAT * [0.0]) + + def assert_exc_string(self, exc_string, expected_exc_name): + exc_lines = exc_string.splitlines() + self.assertGreater(len(exc_lines), 2) + self.assertTrue(exc_lines[0].startswith('Traceback')) + self.assertTrue(exc_lines[-1].startswith(expected_exc_name)) + + def test_print_exc(self): + s = StringIO() + t = timeit.Timer("1/0") + try: + t.timeit() + except: + t.print_exc(s) + self.assert_exc_string(s.getvalue(), 'ZeroDivisionError') + + MAIN_DEFAULT_OUTPUT = "10 loops, best of 3: 1 sec per loop\n" + + def run_main(self, seconds_per_increment=1.0, switches=None, timer=None): + if timer is None: + timer = FakeTimer(seconds_per_increment=seconds_per_increment) + if switches is None: + args = [] + else: + args = switches[:] + args.append(self.fake_stmt) + # timeit.main() modifies sys.path, so save and restore it. + orig_sys_path = sys.path[:] + with captured_stdout() as s: + timeit.main(args=args, _wrap_timer=timer.wrap_timer) + sys.path[:] = orig_sys_path[:] + return s.getvalue() + + def test_main_bad_switch(self): + s = self.run_main(switches=['--bad-switch']) + self.assertEqual(s, dedent("""\ + option --bad-switch not recognized + use -h/--help for command line help + """)) + + def test_main_seconds(self): + s = self.run_main(seconds_per_increment=5.5) + self.assertEqual(s, "10 loops, best of 3: 5.5 sec per loop\n") + + def test_main_milliseconds(self): + s = self.run_main(seconds_per_increment=0.0055) + self.assertEqual(s, "100 loops, best of 3: 5.5 msec per loop\n") + + def test_main_microseconds(self): + s = self.run_main(seconds_per_increment=0.0000025, switches=['-n100']) + self.assertEqual(s, "100 loops, best of 3: 2.5 usec per loop\n") + + def test_main_fixed_iters(self): + s = self.run_main(seconds_per_increment=2.0, switches=['-n35']) + self.assertEqual(s, "35 loops, best of 3: 2 sec per loop\n") + + def test_main_setup(self): + s = self.run_main(seconds_per_increment=2.0, + switches=['-n35', '-s', 'print("CustomSetup")']) + self.assertEqual(s, "CustomSetup\n" * 3 + + "35 loops, best of 3: 2 sec per loop\n") + + def test_main_fixed_reps(self): + s = self.run_main(seconds_per_increment=60.0, switches=['-r9']) + self.assertEqual(s, "10 loops, best of 9: 60 sec per loop\n") + + def test_main_negative_reps(self): + s = self.run_main(seconds_per_increment=60.0, switches=['-r-5']) + self.assertEqual(s, "10 loops, best of 1: 60 sec per loop\n") + + @unittest.skipIf(sys.flags.optimize >= 2, "need __doc__") + def test_main_help(self): + s = self.run_main(switches=['-h']) + self.assertEqual(s, timeit.__doc__) + + def test_main_using_time(self): + fake_timer = FakeTimer() + s = self.run_main(switches=['-t'], timer=fake_timer) + self.assertEqual(s, self.MAIN_DEFAULT_OUTPUT) + self.assertIs(fake_timer.saved_timer, time.time) + + def test_main_using_clock(self): + fake_timer = FakeTimer() + s = self.run_main(switches=['-c'], timer=fake_timer) + self.assertEqual(s, self.MAIN_DEFAULT_OUTPUT) + self.assertIs(fake_timer.saved_timer, time.clock) + + def test_main_verbose(self): + s = self.run_main(switches=['-v']) + self.assertEqual(s, dedent("""\ + 10 loops -> 10 secs + raw times: 10 10 10 + 10 loops, best of 3: 1 sec per loop + """)) + + def test_main_very_verbose(self): + s = self.run_main(seconds_per_increment=0.000050, switches=['-vv']) + self.assertEqual(s, dedent("""\ + 10 loops -> 0.0005 secs + 100 loops -> 0.005 secs + 1000 loops -> 0.05 secs + 10000 loops -> 0.5 secs + raw times: 0.5 0.5 0.5 + 10000 loops, best of 3: 50 usec per loop + """)) + + def test_main_exception(self): + with captured_stderr() as error_stringio: + s = self.run_main(switches=['1/0']) + self.assert_exc_string(error_stringio.getvalue(), 'ZeroDivisionError') + + def test_main_exception_fixed_reps(self): + with captured_stderr() as error_stringio: + s = self.run_main(switches=['-n1', '1/0']) + self.assert_exc_string(error_stringio.getvalue(), 'ZeroDivisionError') + + +def test_main(): + run_unittest(TestTimeit) + +if __name__ == '__main__': + test_main() diff --git a/Lib/timeit.py b/Lib/timeit.py --- a/Lib/timeit.py +++ b/Lib/timeit.py @@ -234,10 +234,10 @@ """Convenience function to create Timer object and call repeat method.""" return Timer(stmt, setup, timer).repeat(repeat, number) -def main(args=None): +def main(args=None, _wrap_timer=None): """Main program, used when run as a script. - The optional argument specifies the command line to be parsed, + The optional 'args' argument specifies the command line to be parsed, defaulting to sys.argv[1:]. The return value is an exit code to be passed to sys.exit(); it @@ -246,6 +246,10 @@ When an exception happens during timing, a traceback is printed to stderr and the return value is 1. Exceptions at other times (including the template compilation) are not caught. + + '_wrap_timer' is an internal interface used for unit testing. If it + is not None, it must be a callable that accepts a timer function + and returns another timer function (used for unit testing). """ if args is None: args = sys.argv[1:] @@ -291,6 +295,8 @@ # directory) import os sys.path.insert(0, os.curdir) + if _wrap_timer is not None: + timer = _wrap_timer(timer) t = Timer(stmt, setup, timer) if number == 0: # determine number so that 0.2 <= total time < 2.0 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -83,6 +83,8 @@ Tests ----- +- Issue #11578: Backported test for the timeit module. + - Issue #22943: bsddb tests are locale independend now. IDLE -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 11:31:54 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 10:31:54 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogVXNlIHRlc3QudGVz?= =?utf-8?q?t=5Fsupport_instead_of_test=2Esupport=2E?= Message-ID: <20150126103141.75800.31666@psf.io> https://hg.python.org/cpython/rev/617c226da195 changeset: 94309:617c226da195 branch: 2.7 parent: 94303:e8db1cbe416b user: Serhiy Storchaka date: Mon Jan 26 12:30:56 2015 +0200 summary: Use test.test_support instead of test.support. files: Lib/test/test_timeit.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_timeit.py b/Lib/test/test_timeit.py --- a/Lib/test/test_timeit.py +++ b/Lib/test/test_timeit.py @@ -5,9 +5,9 @@ import time from textwrap import dedent -from test.support import run_unittest -from test.support import captured_stdout -from test.support import captured_stderr +from test.test_support import run_unittest +from test.test_support import captured_stdout +from test.test_support import captured_stderr # timeit's default number of iterations. DEFAULT_NUMBER = 1000000 -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 12:17:32 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 11:17:32 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319361=3A_JSON_dec?= =?utf-8?q?oder_now_raises_JSONDecodeError_instead_of_ValueError=2E?= Message-ID: <20150126111718.75790.88187@psf.io> https://hg.python.org/cpython/rev/07af9847dbec changeset: 94310:07af9847dbec parent: 94308:3c11b3184b83 user: Serhiy Storchaka date: Mon Jan 26 13:16:30 2015 +0200 summary: Issue #19361: JSON decoder now raises JSONDecodeError instead of ValueError. files: Doc/library/json.rst | 39 ++++++- Doc/whatsnew/3.5.rst | 3 + Lib/json/__init__.py | 7 +- Lib/json/decoder.py | 82 +++++++------- Lib/test/test_json/__init__.py | 4 + Lib/test/test_json/test_decode.py | 8 +- Lib/test/test_json/test_fail.py | 59 +++++++--- Lib/test/test_json/test_scanstring.py | 2 +- Misc/NEWS | 2 + Modules/_json.c | 21 +-- 10 files changed, 144 insertions(+), 83 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -250,7 +250,7 @@ will be passed to the constructor of the class. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. function:: loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) @@ -261,7 +261,7 @@ *encoding* which is ignored and deprecated. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. Encoders and Decoders --------------------- @@ -334,13 +334,16 @@ ``'\n'``, ``'\r'`` and ``'\0'``. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. method:: decode(s) Return the Python representation of *s* (a :class:`str` instance containing a JSON document) + :exc:`JSONDecodeError` will be raised if the given JSON document is not + valid. + .. method:: raw_decode(s) Decode a JSON document from *s* (a :class:`str` beginning with a @@ -469,6 +472,36 @@ mysocket.write(chunk) +Exceptions +---------- + +.. exception:: JSONDecodeError(msg, doc, pos, end=None) + + Subclass of :exc:`ValueError` with the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: doc + + The JSON document being parsed. + + .. attribute:: pos + + The start index of *doc* where parsing failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + + .. versionadded:: 3.5 + + Standard Compliance and Interoperability ---------------------------------------- diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -230,6 +230,9 @@ of dictionaries alphabetically by key. (Contributed by Berker Peksag in :issue:`21650`.) +* JSON decoder now raises :exc:`json.JSONDecodeError` instead of + :exc:`ValueError`. (Contributed by Serhiy Storchaka in :issue:`19361`.) + os -- diff --git a/Lib/json/__init__.py b/Lib/json/__init__.py --- a/Lib/json/__init__.py +++ b/Lib/json/__init__.py @@ -98,12 +98,12 @@ __version__ = '2.0.9' __all__ = [ 'dump', 'dumps', 'load', 'loads', - 'JSONDecoder', 'JSONEncoder', + 'JSONDecoder', 'JSONDecodeError', 'JSONEncoder', ] __author__ = 'Bob Ippolito ' -from .decoder import JSONDecoder +from .decoder import JSONDecoder, JSONDecodeError from .encoder import JSONEncoder _default_encoder = JSONEncoder( @@ -311,7 +311,8 @@ raise TypeError('the JSON object must be str, not {!r}'.format( s.__class__.__name__)) if s.startswith(u'\ufeff'): - raise ValueError("Unexpected UTF-8 BOM (decode using utf-8-sig)") + raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)", + s, 0) if (cls is None and object_hook is None and parse_int is None and parse_float is None and parse_constant is None and object_pairs_hook is None and not kw): diff --git a/Lib/json/decoder.py b/Lib/json/decoder.py --- a/Lib/json/decoder.py +++ b/Lib/json/decoder.py @@ -8,7 +8,7 @@ except ImportError: c_scanstring = None -__all__ = ['JSONDecoder'] +__all__ = ['JSONDecoder', 'JSONDecodeError'] FLAGS = re.VERBOSE | re.MULTILINE | re.DOTALL @@ -17,32 +17,30 @@ NegInf = float('-inf') -def linecol(doc, pos): - if isinstance(doc, bytes): - newline = b'\n' - else: - newline = '\n' - lineno = doc.count(newline, 0, pos) + 1 - if lineno == 1: - colno = pos + 1 - else: - colno = pos - doc.rindex(newline, 0, pos) - return lineno, colno +class JSONDecodeError(ValueError): + """Subclass of ValueError with the following additional properties: + msg: The unformatted error message + doc: The JSON document being parsed + pos: The start index of doc where parsing failed + lineno: The line corresponding to pos + colno: The column corresponding to pos -def errmsg(msg, doc, pos, end=None): - # Note that this function is called from _json - lineno, colno = linecol(doc, pos) - if end is None: - fmt = '{0}: line {1} column {2} (char {3})' - return fmt.format(msg, lineno, colno, pos) - #fmt = '%s: line %d column %d (char %d)' - #return fmt % (msg, lineno, colno, pos) - endlineno, endcolno = linecol(doc, end) - fmt = '{0}: line {1} column {2} - line {3} column {4} (char {5} - {6})' - return fmt.format(msg, lineno, colno, endlineno, endcolno, pos, end) - #fmt = '%s: line %d column %d - line %d column %d (char %d - %d)' - #return fmt % (msg, lineno, colno, endlineno, endcolno, pos, end) + """ + # Note that this exception is used from _json + def __init__(self, msg, doc, pos): + lineno = doc.count('\n', 0, pos) + 1 + colno = pos - doc.rfind('\n', 0, pos) + errmsg = '%s: line %d column %d (char %d)' % (msg, lineno, colno, pos) + ValueError.__init__(self, errmsg) + self.msg = msg + self.doc = doc + self.pos = pos + self.lineno = lineno + self.colno = colno + + def __reduce__(self): + return self.__class__, (self.msg, self.doc, self.pos) _CONSTANTS = { @@ -66,7 +64,7 @@ except ValueError: pass msg = "Invalid \\uXXXX escape" - raise ValueError(errmsg(msg, s, pos)) + raise JSONDecodeError(msg, s, pos) def py_scanstring(s, end, strict=True, _b=BACKSLASH, _m=STRINGCHUNK.match): @@ -84,8 +82,7 @@ while 1: chunk = _m(s, end) if chunk is None: - raise ValueError( - errmsg("Unterminated string starting at", s, begin)) + raise JSONDecodeError("Unterminated string starting at", s, begin) end = chunk.end() content, terminator = chunk.groups() # Content is contains zero or more unescaped string characters @@ -99,22 +96,21 @@ if strict: #msg = "Invalid control character %r at" % (terminator,) msg = "Invalid control character {0!r} at".format(terminator) - raise ValueError(errmsg(msg, s, end)) + raise JSONDecodeError(msg, s, end) else: _append(terminator) continue try: esc = s[end] except IndexError: - raise ValueError( - errmsg("Unterminated string starting at", s, begin)) + raise JSONDecodeError("Unterminated string starting at", s, begin) # If not a unicode escape sequence, must be in the lookup table if esc != 'u': try: char = _b[esc] except KeyError: msg = "Invalid \\escape: {0!r}".format(esc) - raise ValueError(errmsg(msg, s, end)) + raise JSONDecodeError(msg, s, end) end += 1 else: uni = _decode_uXXXX(s, end) @@ -163,8 +159,8 @@ pairs = object_hook(pairs) return pairs, end + 1 elif nextchar != '"': - raise ValueError(errmsg( - "Expecting property name enclosed in double quotes", s, end)) + raise JSONDecodeError( + "Expecting property name enclosed in double quotes", s, end) end += 1 while True: key, end = scanstring(s, end, strict) @@ -174,7 +170,7 @@ if s[end:end + 1] != ':': end = _w(s, end).end() if s[end:end + 1] != ':': - raise ValueError(errmsg("Expecting ':' delimiter", s, end)) + raise JSONDecodeError("Expecting ':' delimiter", s, end) end += 1 try: @@ -188,7 +184,7 @@ try: value, end = scan_once(s, end) except StopIteration as err: - raise ValueError(errmsg("Expecting value", s, err.value)) from None + raise JSONDecodeError("Expecting value", s, err.value) from None pairs_append((key, value)) try: nextchar = s[end] @@ -202,13 +198,13 @@ if nextchar == '}': break elif nextchar != ',': - raise ValueError(errmsg("Expecting ',' delimiter", s, end - 1)) + raise JSONDecodeError("Expecting ',' delimiter", s, end - 1) end = _w(s, end).end() nextchar = s[end:end + 1] end += 1 if nextchar != '"': - raise ValueError(errmsg( - "Expecting property name enclosed in double quotes", s, end - 1)) + raise JSONDecodeError( + "Expecting property name enclosed in double quotes", s, end - 1) if object_pairs_hook is not None: result = object_pairs_hook(pairs) return result, end @@ -232,7 +228,7 @@ try: value, end = scan_once(s, end) except StopIteration as err: - raise ValueError(errmsg("Expecting value", s, err.value)) from None + raise JSONDecodeError("Expecting value", s, err.value) from None _append(value) nextchar = s[end:end + 1] if nextchar in _ws: @@ -242,7 +238,7 @@ if nextchar == ']': break elif nextchar != ',': - raise ValueError(errmsg("Expecting ',' delimiter", s, end - 1)) + raise JSONDecodeError("Expecting ',' delimiter", s, end - 1) try: if s[end] in _ws: end += 1 @@ -343,7 +339,7 @@ obj, end = self.raw_decode(s, idx=_w(s, 0).end()) end = _w(s, end).end() if end != len(s): - raise ValueError(errmsg("Extra data", s, end, len(s))) + raise JSONDecodeError("Extra data", s, end) return obj def raw_decode(self, s, idx=0): @@ -358,5 +354,5 @@ try: obj, end = self.scan_once(s, idx) except StopIteration as err: - raise ValueError(errmsg("Expecting value", s, err.value)) from None + raise JSONDecodeError("Expecting value", s, err.value) from None return obj, end diff --git a/Lib/test/test_json/__init__.py b/Lib/test/test_json/__init__.py --- a/Lib/test/test_json/__init__.py +++ b/Lib/test/test_json/__init__.py @@ -9,12 +9,15 @@ # import json with and without accelerations cjson = support.import_fresh_module('json', fresh=['_json']) pyjson = support.import_fresh_module('json', blocked=['_json']) +# JSONDecodeError is cached inside the _json module +cjson.JSONDecodeError = cjson.decoder.JSONDecodeError = json.JSONDecodeError # create two base classes that will be used by the other tests class PyTest(unittest.TestCase): json = pyjson loads = staticmethod(pyjson.loads) dumps = staticmethod(pyjson.dumps) + JSONDecodeError = staticmethod(pyjson.JSONDecodeError) @unittest.skipUnless(cjson, 'requires _json') class CTest(unittest.TestCase): @@ -22,6 +25,7 @@ json = cjson loads = staticmethod(cjson.loads) dumps = staticmethod(cjson.dumps) + JSONDecodeError = staticmethod(cjson.JSONDecodeError) # test PyTest and CTest checking if the functions come from the right module class TestPyTest(PyTest): diff --git a/Lib/test/test_json/test_decode.py b/Lib/test/test_json/test_decode.py --- a/Lib/test/test_json/test_decode.py +++ b/Lib/test/test_json/test_decode.py @@ -63,12 +63,12 @@ def test_extra_data(self): s = '[1, 2, 3]5' msg = 'Extra data' - self.assertRaisesRegex(ValueError, msg, self.loads, s) + self.assertRaisesRegex(self.JSONDecodeError, msg, self.loads, s) def test_invalid_escape(self): s = '["abc\\y"]' msg = 'escape' - self.assertRaisesRegex(ValueError, msg, self.loads, s) + self.assertRaisesRegex(self.JSONDecodeError, msg, self.loads, s) def test_invalid_input_type(self): msg = 'the JSON object must be str' @@ -80,10 +80,10 @@ def test_string_with_utf8_bom(self): # see #18958 bom_json = "[1,2,3]".encode('utf-8-sig').decode('utf-8') - with self.assertRaises(ValueError) as cm: + with self.assertRaises(self.JSONDecodeError) as cm: self.loads(bom_json) self.assertIn('BOM', str(cm.exception)) - with self.assertRaises(ValueError) as cm: + with self.assertRaises(self.JSONDecodeError) as cm: self.json.load(StringIO(bom_json)) self.assertIn('BOM', str(cm.exception)) # make sure that the BOM is not detected in the middle of a string diff --git a/Lib/test/test_json/test_fail.py b/Lib/test/test_json/test_fail.py --- a/Lib/test/test_json/test_fail.py +++ b/Lib/test/test_json/test_fail.py @@ -87,7 +87,7 @@ continue try: self.loads(doc) - except ValueError: + except self.JSONDecodeError: pass else: self.fail("Expected failure for fail{0}.json: {1!r}".format(idx, doc)) @@ -124,10 +124,16 @@ ('"spam', 'Unterminated string starting at', 0), ] for data, msg, idx in test_cases: - self.assertRaisesRegex(ValueError, - r'^{0}: line 1 column {1} \(char {2}\)'.format( - re.escape(msg), idx + 1, idx), - self.loads, data) + with self.assertRaises(self.JSONDecodeError) as cm: + self.loads(data) + err = cm.exception + self.assertEqual(err.msg, msg) + self.assertEqual(err.pos, idx) + self.assertEqual(err.lineno, 1) + self.assertEqual(err.colno, idx + 1) + self.assertEqual(str(err), + '%s: line 1 column %d (char %d)' % + (msg, idx + 1, idx)) def test_unexpected_data(self): test_cases = [ @@ -154,10 +160,16 @@ ('{"spam":42,}', 'Expecting property name enclosed in double quotes', 11), ] for data, msg, idx in test_cases: - self.assertRaisesRegex(ValueError, - r'^{0}: line 1 column {1} \(char {2}\)'.format( - re.escape(msg), idx + 1, idx), - self.loads, data) + with self.assertRaises(self.JSONDecodeError) as cm: + self.loads(data) + err = cm.exception + self.assertEqual(err.msg, msg) + self.assertEqual(err.pos, idx) + self.assertEqual(err.lineno, 1) + self.assertEqual(err.colno, idx + 1) + self.assertEqual(str(err), + '%s: line 1 column %d (char %d)' % + (msg, idx + 1, idx)) def test_extra_data(self): test_cases = [ @@ -171,11 +183,16 @@ ('"spam",42', 'Extra data', 6), ] for data, msg, idx in test_cases: - self.assertRaisesRegex(ValueError, - r'^{0}: line 1 column {1} - line 1 column {2}' - r' \(char {3} - {4}\)'.format( - re.escape(msg), idx + 1, len(data) + 1, idx, len(data)), - self.loads, data) + with self.assertRaises(self.JSONDecodeError) as cm: + self.loads(data) + err = cm.exception + self.assertEqual(err.msg, msg) + self.assertEqual(err.pos, idx) + self.assertEqual(err.lineno, 1) + self.assertEqual(err.colno, idx + 1) + self.assertEqual(str(err), + '%s: line 1 column %d (char %d)' % + (msg, idx + 1, idx)) def test_linecol(self): test_cases = [ @@ -185,10 +202,16 @@ ('\n \n\n !', 4, 6, 10), ] for data, line, col, idx in test_cases: - self.assertRaisesRegex(ValueError, - r'^Expecting value: line {0} column {1}' - r' \(char {2}\)$'.format(line, col, idx), - self.loads, data) + with self.assertRaises(self.JSONDecodeError) as cm: + self.loads(data) + err = cm.exception + self.assertEqual(err.msg, 'Expecting value') + self.assertEqual(err.pos, idx) + self.assertEqual(err.lineno, line) + self.assertEqual(err.colno, col) + self.assertEqual(str(err), + 'Expecting value: line %s column %d (char %d)' % + (line, col, idx)) class TestPyFail(TestFail, PyTest): pass class TestCFail(TestFail, CTest): pass diff --git a/Lib/test/test_json/test_scanstring.py b/Lib/test/test_json/test_scanstring.py --- a/Lib/test/test_json/test_scanstring.py +++ b/Lib/test/test_json/test_scanstring.py @@ -129,7 +129,7 @@ '"\\ud834\\u0X20"', ] for s in bad_escapes: - with self.assertRaises(ValueError, msg=s): + with self.assertRaises(self.JSONDecodeError, msg=s): scanstring(s, 1, True) def test_overflow(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -218,6 +218,8 @@ Library ------- +- Issue #19361: JSON decoder now raises JSONDecodeError instead of ValueError. + - Issue #18518: timeit now rejects statements which can't be compiled outside a function or a loop (e.g. "return" or "break"). diff --git a/Modules/_json.c b/Modules/_json.c --- a/Modules/_json.c +++ b/Modules/_json.c @@ -312,23 +312,22 @@ static void raise_errmsg(char *msg, PyObject *s, Py_ssize_t end) { - /* Use the Python function json.decoder.errmsg to raise a nice - looking ValueError exception */ - static PyObject *errmsg_fn = NULL; - PyObject *pymsg; - if (errmsg_fn == NULL) { + /* Use JSONDecodeError exception to raise a nice looking ValueError subclass */ + static PyObject *JSONDecodeError = NULL; + PyObject *exc; + if (JSONDecodeError == NULL) { PyObject *decoder = PyImport_ImportModule("json.decoder"); if (decoder == NULL) return; - errmsg_fn = PyObject_GetAttrString(decoder, "errmsg"); + JSONDecodeError = PyObject_GetAttrString(decoder, "JSONDecodeError"); Py_DECREF(decoder); - if (errmsg_fn == NULL) + if (JSONDecodeError == NULL) return; } - pymsg = PyObject_CallFunction(errmsg_fn, "(zOn)", msg, s, end); - if (pymsg) { - PyErr_SetObject(PyExc_ValueError, pymsg); - Py_DECREF(pymsg); + exc = PyObject_CallFunction(JSONDecodeError, "(zOn)", msg, s, end); + if (exc) { + PyErr_SetObject(JSONDecodeError, exc); + Py_DECREF(exc); } } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 13:02:20 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 12:02:20 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <20150126120214.55128.95204@psf.io> https://hg.python.org/cpython/rev/d72a293f1b86 changeset: 94313:d72a293f1b86 parent: 94310:07af9847dbec parent: 94312:680b47c96e08 user: Serhiy Storchaka date: Mon Jan 26 13:47:52 2015 +0200 summary: Null merge files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 13:02:20 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 12:02:20 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2314099=3A_Restored?= =?utf-8?q?_support_of_writing_ZIP_files_to_tellable_but?= Message-ID: <20150126120218.87117.31359@psf.io> https://hg.python.org/cpython/rev/9cbf9f96920d changeset: 94315:9cbf9f96920d user: Serhiy Storchaka date: Mon Jan 26 14:01:27 2015 +0200 summary: Issue #14099: Restored support of writing ZIP files to tellable but non-seekable streams. files: Lib/test/test_zipfile.py | 28 ++++++++++++++++++++++++++++ Lib/zipfile.py | 16 +++++++++++++--- Misc/NEWS | 3 +++ 3 files changed, 44 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -1668,6 +1668,34 @@ compression = zipfile.ZIP_LZMA +# Privide the tell() method but not seek() +class Tellable: + def __init__(self, fp): + self.fp = fp + self.offset = 0 + + def write(self, data): + self.offset += self.fp.write(data) + + def tell(self): + return self.offset + + def flush(self): + pass + +class UnseekableTests(unittest.TestCase): + def test_writestr_tellable(self): + f = io.BytesIO() + with zipfile.ZipFile(Tellable(f), 'w', zipfile.ZIP_STORED) as zipfp: + zipfp.writestr('ones', b'111') + zipfp.writestr('twos', b'222') + with zipfile.ZipFile(f, mode='r') as zipf: + with zipf.open('ones') as zopen: + self.assertEqual(zopen.read(), b'111') + with zipf.open('twos') as zopen: + self.assertEqual(zopen.read(), b'222') + + @requires_zlib class TestsWithMultipleOpens(unittest.TestCase): @classmethod diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1504,7 +1504,14 @@ zinfo.file_size = len(data) # Uncompressed size with self._lock: - self.fp.seek(self.start_dir, 0) + try: + self.fp.seek(self.start_dir) + except (AttributeError, io.UnsupportedOperation): + # Some file-like objects can provide tell() but not seek() + pass + zinfo.header_offset = self.fp.tell() # Start of header data + if compress_type is not None: + zinfo.compress_type = compress_type zinfo.header_offset = self.fp.tell() # Start of header data if compress_type is not None: zinfo.compress_type = compress_type @@ -1550,7 +1557,11 @@ try: if self.mode in ("w", "a") and self._didModify: # write ending records with self._lock: - self.fp.seek(self.start_dir, 0) + try: + self.fp.seek(self.start_dir) + except (AttributeError, io.UnsupportedOperation): + # Some file-like objects can provide tell() but not seek() + pass self._write_end_record() finally: fp = self.fp @@ -1558,7 +1569,6 @@ self._fpclose(fp) def _write_end_record(self): - self.fp.seek(self.start_dir, 0) for zinfo in self.filelist: # write central directory dt = zinfo.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -218,6 +218,9 @@ Library ------- +- Issue #14099: Restored support of writing ZIP files to tellable but + non-seekable streams. + - Issue #14099: Writing to ZipFile and reading multiple ZipExtFiles is threadsafe now. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 13:02:20 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 12:02:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE0MDk5?= =?utf-8?q?=3A_Backout_changeset_e5bb3044402b_=28except_adapted_tests=29?= =?utf-8?q?=2E?= Message-ID: <20150126120213.55120.89883@psf.io> https://hg.python.org/cpython/rev/680b47c96e08 changeset: 94312:680b47c96e08 branch: 3.4 parent: 94306:146fe233ecc0 user: Serhiy Storchaka date: Mon Jan 26 13:45:39 2015 +0200 summary: Issue #14099: Backout changeset e5bb3044402b (except adapted tests). files: Doc/library/zipfile.rst | 10 +- Lib/test/test_zipfile.py | 122 ++++++++++++-------------- Lib/zipfile.py | 103 ++++++++-------------- Misc/NEWS | 5 - 4 files changed, 105 insertions(+), 135 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -219,8 +219,14 @@ .. note:: - Objects returned by :meth:`.open` can operate independently of the - ZipFile. + If the ZipFile was created by passing in a file-like object as the first + argument to the constructor, then the object returned by :meth:`.open` shares the + ZipFile's file pointer. Under these circumstances, the object returned by + :meth:`.open` should not be used after any additional operations are performed + on the ZipFile object. If the ZipFile was created by passing in a string (the + filename) as the first argument to the constructor, then :meth:`.open` will + create a new file object that will be held by the ZipExtFile, allowing it to + operate independently of the ZipFile. .. note:: diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -1653,52 +1653,22 @@ def test_same_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - for f in get_files(self): - self.make_test_archive(f) - with zipfile.ZipFile(f, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read() - data2 += zopen2.read() - self.assertEqual(data1, data2) - self.assertEqual(data1, self.data1) + self.make_test_archive(TESTFN2) + with zipfile.ZipFile(TESTFN2, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, data2) + self.assertEqual(data1, self.data1) def test_different_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - for f in get_files(self): - self.make_test_archive(f) - with zipfile.ZipFile(f, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read() - data2 += zopen2.read() - self.assertEqual(data1, self.data1) - self.assertEqual(data2, self.data2) - - def test_interleaved(self): - # Verify that (when the ZipFile is in control of creating file objects) - # multiple open() calls can be made without interfering with each other. - for f in get_files(self): - self.make_test_archive(f) - with zipfile.ZipFile(f, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read() - data2 += zopen2.read() - self.assertEqual(data1, self.data1) - self.assertEqual(data2, self.data2) - - def test_read_after_close(self): - for f in get_files(self): - self.make_test_archive(f) - with contextlib.ExitStack() as stack: - with zipfile.ZipFile(f, 'r') as zipf: - zopen1 = stack.enter_context(zipf.open('ones')) - zopen2 = stack.enter_context(zipf.open('twos')) + self.make_test_archive(TESTFN2) + with zipfile.ZipFile(TESTFN2, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: data1 = zopen1.read(500) data2 = zopen2.read(500) data1 += zopen1.read() @@ -1706,32 +1676,56 @@ self.assertEqual(data1, self.data1) self.assertEqual(data2, self.data2) - def test_read_after_write(self): - for f in get_files(self): - with zipfile.ZipFile(f, 'w', zipfile.ZIP_DEFLATED) as zipf: - zipf.writestr('ones', self.data1) - zipf.writestr('twos', self.data2) - with zipf.open('ones') as zopen1: - data1 = zopen1.read(500) - self.assertEqual(data1, self.data1[:500]) - with zipfile.ZipFile(f, 'r') as zipf: - data1 = zipf.read('ones') - data2 = zipf.read('twos') + def test_interleaved(self): + # Verify that (when the ZipFile is in control of creating file objects) + # multiple open() calls can be made without interfering with each other. + self.make_test_archive(TESTFN2) + with zipfile.ZipFile(TESTFN2, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() self.assertEqual(data1, self.data1) self.assertEqual(data2, self.data2) + def test_read_after_close(self): + self.make_test_archive(TESTFN2) + with contextlib.ExitStack() as stack: + with zipfile.ZipFile(TESTFN2, 'r') as zipf: + zopen1 = stack.enter_context(zipf.open('ones')) + zopen2 = stack.enter_context(zipf.open('twos')) + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_read_after_write(self): + with zipfile.ZipFile(TESTFN2, 'w', zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + zipf.writestr('twos', self.data2) + with zipf.open('ones') as zopen1: + data1 = zopen1.read(500) + self.assertEqual(data1, self.data1[:500]) + with zipfile.ZipFile(TESTFN2, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + def test_write_after_read(self): - for f in get_files(self): - with zipfile.ZipFile(f, "w", zipfile.ZIP_DEFLATED) as zipf: - zipf.writestr('ones', self.data1) - with zipf.open('ones') as zopen1: - zopen1.read(500) - zipf.writestr('twos', self.data2) - with zipfile.ZipFile(f, 'r') as zipf: - data1 = zipf.read('ones') - data2 = zipf.read('twos') - self.assertEqual(data1, self.data1) - self.assertEqual(data2, self.data2) + with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + with zipf.open('ones') as zopen1: + zopen1.read(500) + zipf.writestr('twos', self.data2) + with zipfile.ZipFile(TESTFN2, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) def test_many_opens(self): # Verify that read() and open() promptly close the file descriptor, diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -624,25 +624,6 @@ raise NotImplementedError("compression type %d" % (compress_type,)) -class _SharedFile: - def __init__(self, file, pos, close): - self._file = file - self._pos = pos - self._close = close - - def read(self, n=-1): - self._file.seek(self._pos) - data = self._file.read(n) - self._pos = self._file.tell() - return data - - def close(self): - if self._file is not None: - fileobj = self._file - self._file = None - self._close(fileobj) - - class ZipExtFile(io.BufferedIOBase): """File-like object for reading an archive member. Is returned by ZipFile.open(). @@ -928,7 +909,7 @@ self.NameToInfo = {} # Find file info given name self.filelist = [] # List of ZipInfo instances for archive self.compression = compression # Method of compression - self.mode = mode + self.mode = key = mode.replace('b', '')[0] self.pwd = None self._comment = b'' @@ -937,33 +918,28 @@ # No, it's a filename self._filePassed = 0 self.filename = file - modeDict = {'r' : 'rb', 'w': 'w+b', 'a' : 'r+b', - 'r+b': 'w+b', 'w+b': 'wb'} - filemode = modeDict[mode] - while True: - try: - self.fp = io.open(file, filemode) - except OSError: - if filemode in modeDict: - filemode = modeDict[filemode] - continue + modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'} + try: + self.fp = io.open(file, modeDict[mode]) + except OSError: + if mode == 'a': + mode = key = 'w' + self.fp = io.open(file, modeDict[mode]) + else: raise - break else: self._filePassed = 1 self.fp = file self.filename = getattr(file, 'name', None) - self._fileRefCnt = 1 try: - if mode == 'r': + if key == 'r': self._RealGetContents() - elif mode == 'w': + elif key == 'w': # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True - self.start_dir = 0 - elif mode == 'a': + elif key == 'a': try: # See if file is a zip file self._RealGetContents() @@ -976,13 +952,13 @@ # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True - self.start_dir = self.fp.tell() else: raise RuntimeError('Mode must be "r", "w" or "a"') except: fp = self.fp self.fp = None - self._fpclose(fp) + if not self._filePassed: + fp.close() raise def __enter__(self): @@ -1155,17 +1131,23 @@ raise RuntimeError( "Attempt to read ZIP archive that was already closed") - # Make sure we have an info object - if isinstance(name, ZipInfo): - # 'name' is already an info object - zinfo = name + # Only open a new file for instances where we were not + # given a file object in the constructor + if self._filePassed: + zef_file = self.fp else: - # Get info object for name - zinfo = self.getinfo(name) + zef_file = io.open(self.filename, 'rb') - self._fileRefCnt += 1 - zef_file = _SharedFile(self.fp, zinfo.header_offset, self._fpclose) try: + # Make sure we have an info object + if isinstance(name, ZipInfo): + # 'name' is already an info object + zinfo = name + else: + # Get info object for name + zinfo = self.getinfo(name) + zef_file.seek(zinfo.header_offset, 0) + # Skip the file header: fheader = zef_file.read(sizeFileHeader) if len(fheader) != sizeFileHeader: @@ -1224,9 +1206,11 @@ if h[11] != check_byte: raise RuntimeError("Bad password for file", name) - return ZipExtFile(zef_file, mode, zinfo, zd, True) + return ZipExtFile(zef_file, mode, zinfo, zd, + close_fileobj=not self._filePassed) except: - zef_file.close() + if not self._filePassed: + zef_file.close() raise def extract(self, member, path=None, pwd=None): @@ -1360,7 +1344,6 @@ zinfo.file_size = st.st_size zinfo.flag_bits = 0x00 - self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header bytes if zinfo.compress_type == ZIP_LZMA: # Compressed data includes an end-of-stream (EOS) marker @@ -1377,7 +1360,6 @@ self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo self.fp.write(zinfo.FileHeader(False)) - self.start_dir = self.fp.tell() return cmpr = _get_compressor(zinfo.compress_type) @@ -1416,10 +1398,10 @@ raise RuntimeError('Compressed size larger than uncompressed size') # Seek backwards and write file header (which will now include # correct CRC and file sizes) - self.start_dir = self.fp.tell() # Preserve current position in file + position = self.fp.tell() # Preserve current position in file self.fp.seek(zinfo.header_offset, 0) self.fp.write(zinfo.FileHeader(zip64)) - self.fp.seek(self.start_dir, 0) + self.fp.seek(position, 0) self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1448,7 +1430,6 @@ "Attempt to write to ZIP archive that was already closed") zinfo.file_size = len(data) # Uncompressed size - self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header data if compress_type is not None: zinfo.compress_type = compress_type @@ -1477,7 +1458,6 @@ self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size, zinfo.file_size)) self.fp.flush() - self.start_dir = self.fp.tell() self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1493,7 +1473,7 @@ try: if self.mode in ("w", "a") and self._didModify: # write ending records - self.fp.seek(self.start_dir, 0) + pos1 = self.fp.tell() for zinfo in self.filelist: # write central directory dt = zinfo.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -1559,8 +1539,8 @@ pos2 = self.fp.tell() # Write end-of-zip-archive record centDirCount = len(self.filelist) - centDirSize = pos2 - self.start_dir - centDirOffset = self.start_dir + centDirSize = pos2 - pos1 + centDirOffset = pos1 requires_zip64 = None if centDirCount > ZIP_FILECOUNT_LIMIT: requires_zip64 = "Files count" @@ -1596,13 +1576,8 @@ finally: fp = self.fp self.fp = None - self._fpclose(fp) - - def _fpclose(self, fp): - assert self._fileRefCnt > 0 - self._fileRefCnt -= 1 - if not self._fileRefCnt and not self._filePassed: - fp.close() + if not self._filePassed: + fp.close() class PyZipFile(ZipFile): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -132,11 +132,6 @@ - Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode will return. This resolves CVE-2013-1753. -- Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects - returned by ZipFile.open() can now operate independently of the ZipFile even - if the ZipFile was created by passing in a file-like object as the first - argument to the constructor. - - Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is asked to compile a source file containing multiple dots in the source file name. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 13:02:20 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 12:02:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0MDk5?= =?utf-8?q?=3A_Backout_changeset_c2c4cde55f6f_=28except_adapted_tests=29?= =?utf-8?q?=2E?= Message-ID: <20150126120213.55102.58269@psf.io> https://hg.python.org/cpython/rev/ae42c4576438 changeset: 94311:ae42c4576438 branch: 2.7 parent: 94309:617c226da195 user: Serhiy Storchaka date: Mon Jan 26 13:45:04 2015 +0200 summary: Issue #14099: Backout changeset c2c4cde55f6f (except adapted tests). files: Doc/library/zipfile.rst | 10 +- Lib/test/test_zipfile.py | 140 +++++++++++--------------- Lib/zipfile.py | 106 ++++++++------------ Misc/NEWS | 5 - 4 files changed, 111 insertions(+), 150 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -186,8 +186,14 @@ .. note:: - Objects returned by :meth:`.open` can operate independently of the - ZipFile. + If the ZipFile was created by passing in a file-like object as the first + argument to the constructor, then the object returned by :meth:`.open` shares the + ZipFile's file pointer. Under these circumstances, the object returned by + :meth:`.open` should not be used after any additional operations are performed + on the ZipFile object. If the ZipFile was created by passing in a string (the + filename) as the first argument to the constructor, then :meth:`.open` will + create a new file object that will be held by the ZipExtFile, allowing it to + operate independently of the ZipFile. .. note:: diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -36,20 +36,8 @@ ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] def getrandbytes(size): - return getrandbits(8 * size).to_bytes(size, 'little') - -def getrandbytes(size): return bytes(bytearray.fromhex('%0*x' % (2 * size, getrandbits(8 * size)))) -def get_files(test): - yield TESTFN2 - with TemporaryFile() as f: - yield f - test.assertFalse(f.closed) - with io.BytesIO() as f: - yield f - test.assertFalse(f.closed) - class TestsWithSourceFile(unittest.TestCase): def setUp(self): self.line_gen = ["Zipfile test line %d. random float: %f" % (i, random()) @@ -1427,91 +1415,85 @@ def test_same_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - for f in get_files(self): - self.make_test_archive(f) - with zipfile.ZipFile(f, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read() - data2 += zopen2.read() - self.assertEqual(data1, data2) - self.assertEqual(data1, self.data1) + self.make_test_archive(TESTFN2) + with zipfile.ZipFile(TESTFN2, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, data2) + self.assertEqual(data1, self.data1) def test_different_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - for f in get_files(self): - self.make_test_archive(f) - with zipfile.ZipFile(f, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read() - data2 += zopen2.read() - self.assertEqual(data1, self.data1) - self.assertEqual(data2, self.data2) + self.make_test_archive(TESTFN2) + with zipfile.ZipFile(TESTFN2, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) def test_interleaved(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - for f in get_files(self): - self.make_test_archive(f) - with zipfile.ZipFile(f, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read() - data2 += zopen2.read() - self.assertEqual(data1, self.data1) - self.assertEqual(data2, self.data2) - - def test_read_after_close(self): - for f in get_files(self): - self.make_test_archive(f) - zopen1 = zopen2 = None - try: - with zipfile.ZipFile(f, 'r') as zipf: - zopen1 = zipf.open('ones') - zopen2 = zipf.open('twos') + self.make_test_archive(TESTFN2) + with zipfile.ZipFile(TESTFN2, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: data1 = zopen1.read(500) data2 = zopen2.read(500) data1 += zopen1.read() data2 += zopen2.read() - finally: - if zopen1: - zopen1.close() - if zopen2: - zopen2.close() self.assertEqual(data1, self.data1) self.assertEqual(data2, self.data2) + def test_read_after_close(self): + self.make_test_archive(TESTFN2) + zopen1 = zopen2 = None + try: + with zipfile.ZipFile(TESTFN2, 'r') as zipf: + zopen1 = zipf.open('ones') + zopen2 = zipf.open('twos') + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + finally: + if zopen1: + zopen1.close() + if zopen2: + zopen2.close() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + def test_read_after_write(self): - for f in get_files(self): - with zipfile.ZipFile(f, 'w', zipfile.ZIP_DEFLATED) as zipf: - zipf.writestr('ones', self.data1) - zipf.writestr('twos', self.data2) - with zipf.open('ones') as zopen1: - data1 = zopen1.read(500) - self.assertEqual(data1, self.data1[:500]) - with zipfile.ZipFile(f, 'r') as zipf: - data1 = zipf.read('ones') - data2 = zipf.read('twos') - self.assertEqual(data1, self.data1) - self.assertEqual(data2, self.data2) + with zipfile.ZipFile(TESTFN2, 'w', zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + zipf.writestr('twos', self.data2) + with zipf.open('ones') as zopen1: + data1 = zopen1.read(500) + self.assertEqual(data1, self.data1[:500]) + with zipfile.ZipFile(TESTFN2, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) def test_write_after_read(self): - for f in get_files(self): - with zipfile.ZipFile(f, "w", zipfile.ZIP_DEFLATED) as zipf: - zipf.writestr('ones', self.data1) - with zipf.open('ones') as zopen1: - zopen1.read(500) - zipf.writestr('twos', self.data2) - with zipfile.ZipFile(f, 'r') as zipf: - data1 = zipf.read('ones') - data2 = zipf.read('twos') - self.assertEqual(data1, self.data1) - self.assertEqual(data2, self.data2) + with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + with zipf.open('ones') as zopen1: + zopen1.read(500) + zipf.writestr('twos', self.data2) + with zipfile.ZipFile(TESTFN2, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) def test_many_opens(self): # Verify that read() and open() promptly close the file descriptor, diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -498,25 +498,6 @@ } -class _SharedFile: - def __init__(self, file, pos, close): - self._file = file - self._pos = pos - self._close = close - - def read(self, n=-1): - self._file.seek(self._pos) - data = self._file.read(n) - self._pos = self._file.tell() - return data - - def close(self): - if self._file is not None: - fileobj = self._file - self._file = None - self._close(fileobj) - - class ZipExtFile(io.BufferedIOBase): """File-like object for reading an archive member. Is returned by ZipFile.open(). @@ -762,7 +743,7 @@ self.NameToInfo = {} # Find file info given name self.filelist = [] # List of ZipInfo instances for archive self.compression = compression # Method of compression - self.mode = mode + self.mode = key = mode.replace('b', '')[0] self.pwd = None self._comment = '' @@ -770,33 +751,28 @@ if isinstance(file, basestring): self._filePassed = 0 self.filename = file - modeDict = {'r' : 'rb', 'w': 'w+b', 'a' : 'r+b', - 'r+b': 'w+b', 'w+b': 'wb'} - filemode = modeDict[mode] - while True: - try: - self.fp = io.open(file, filemode) - except IOError: - if filemode in modeDict: - filemode = modeDict[filemode] - continue + modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'} + try: + self.fp = open(file, modeDict[mode]) + except IOError: + if mode == 'a': + mode = key = 'w' + self.fp = open(file, modeDict[mode]) + else: raise - break else: self._filePassed = 1 self.fp = file self.filename = getattr(file, 'name', None) - self._fileRefCnt = 1 try: - if mode == 'r': + if key == 'r': self._RealGetContents() - elif mode == 'w': + elif key == 'w': # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True - self.start_dir = 0 - elif mode == 'a': + elif key == 'a': try: # See if file is a zip file self._RealGetContents() @@ -809,13 +785,13 @@ # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True - self.start_dir = self.fp.tell() else: raise RuntimeError('Mode must be "r", "w" or "a"') except: fp = self.fp self.fp = None - self._fpclose(fp) + if not self._filePassed: + fp.close() raise def __enter__(self): @@ -966,17 +942,26 @@ raise RuntimeError, \ "Attempt to read ZIP archive that was already closed" - # Make sure we have an info object - if isinstance(name, ZipInfo): - # 'name' is already an info object - zinfo = name + # Only open a new file for instances where we were not + # given a file object in the constructor + if self._filePassed: + zef_file = self.fp + should_close = False else: - # Get info object for name - zinfo = self.getinfo(name) + zef_file = open(self.filename, 'rb') + should_close = True - self._fileRefCnt += 1 - zef_file = _SharedFile(self.fp, zinfo.header_offset, self._fpclose) try: + # Make sure we have an info object + if isinstance(name, ZipInfo): + # 'name' is already an info object + zinfo = name + else: + # Get info object for name + zinfo = self.getinfo(name) + + zef_file.seek(zinfo.header_offset, 0) + # Skip the file header: fheader = zef_file.read(sizeFileHeader) if len(fheader) != sizeFileHeader: @@ -1021,9 +1006,11 @@ if ord(h[11]) != check_byte: raise RuntimeError("Bad password for file", name) - return ZipExtFile(zef_file, mode, zinfo, zd, True) + return ZipExtFile(zef_file, mode, zinfo, zd, + close_fileobj=should_close) except: - zef_file.close() + if should_close: + zef_file.close() raise def extract(self, member, path=None, pwd=None): @@ -1154,7 +1141,6 @@ zinfo.file_size = st.st_size zinfo.flag_bits = 0x00 - self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header bytes self._writecheck(zinfo) @@ -1168,7 +1154,6 @@ self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo self.fp.write(zinfo.FileHeader(False)) - self.start_dir = self.fp.tell() return with open(filename, "rb") as fp: @@ -1211,10 +1196,10 @@ raise RuntimeError('Compressed size larger than uncompressed size') # Seek backwards and write file header (which will now include # correct CRC and file sizes) - self.start_dir = self.fp.tell() # Preserve current position in file + position = self.fp.tell() # Preserve current position in file self.fp.seek(zinfo.header_offset, 0) self.fp.write(zinfo.FileHeader(zip64)) - self.fp.seek(self.start_dir, 0) + self.fp.seek(position, 0) self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1243,7 +1228,6 @@ zinfo.compress_type = compress_type zinfo.file_size = len(bytes) # Uncompressed size - self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header bytes self._writecheck(zinfo) self._didModify = True @@ -1267,7 +1251,6 @@ self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size, zinfo.file_size)) self.fp.flush() - self.start_dir = self.fp.tell() self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1283,7 +1266,7 @@ try: if self.mode in ("w", "a") and self._didModify: # write ending records - self.fp.seek(self.start_dir, 0) + pos1 = self.fp.tell() for zinfo in self.filelist: # write central directory dt = zinfo.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -1346,8 +1329,8 @@ pos2 = self.fp.tell() # Write end-of-zip-archive record centDirCount = len(self.filelist) - centDirSize = pos2 - self.start_dir - centDirOffset = self.start_dir + centDirSize = pos2 - pos1 + centDirOffset = pos1 requires_zip64 = None if centDirCount > ZIP_FILECOUNT_LIMIT: requires_zip64 = "Files count" @@ -1383,13 +1366,8 @@ finally: fp = self.fp self.fp = None - self._fpclose(fp) - - def _fpclose(self, fp): - assert self._fileRefCnt > 0 - self._fileRefCnt -= 1 - if not self._fileRefCnt and not self._filePassed: - fp.close() + if not self._filePassed: + fp.close() class PyZipFile(ZipFile): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -53,11 +53,6 @@ - Issue #23016: A warning no longer produces an AttributeError when sys.stderr is None. -- Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects - returned by ZipFile.open() can now operate independently of the ZipFile even - if the ZipFile was created by passing in a file-like object as the first - argument to the constructor. - - Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. Original patch by Martin Panter. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 13:02:20 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 26 Jan 2015 12:02:20 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2314099=3A_Writing_?= =?utf-8?q?to_ZipFile_and_reading_multiple_ZipExtFiles_is?= Message-ID: <20150126120214.118226.24927@psf.io> https://hg.python.org/cpython/rev/4973ccd46e32 changeset: 94314:4973ccd46e32 user: Serhiy Storchaka date: Mon Jan 26 13:53:38 2015 +0200 summary: Issue #14099: Writing to ZipFile and reading multiple ZipExtFiles is threadsafe now. files: Lib/zipfile.py | 401 +++++++++++++++++++----------------- Misc/NEWS | 3 + 2 files changed, 209 insertions(+), 195 deletions(-) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -13,6 +13,7 @@ import shutil import struct import binascii +import threading try: @@ -647,16 +648,18 @@ class _SharedFile: - def __init__(self, file, pos, close): + def __init__(self, file, pos, close, lock): self._file = file self._pos = pos self._close = close + self._lock = lock def read(self, n=-1): - self._file.seek(self._pos) - data = self._file.read(n) - self._pos = self._file.tell() - return data + with self._lock: + self._file.seek(self._pos) + data = self._file.read(n) + self._pos = self._file.tell() + return data def close(self): if self._file is not None: @@ -990,6 +993,7 @@ self.fp = file self.filename = getattr(file, 'name', None) self._fileRefCnt = 1 + self._lock = threading.RLock() try: if mode == 'r': @@ -1214,7 +1218,7 @@ zinfo = self.getinfo(name) self._fileRefCnt += 1 - zef_file = _SharedFile(self.fp, zinfo.header_offset, self._fpclose) + zef_file = _SharedFile(self.fp, zinfo.header_offset, self._fpclose, self._lock) try: # Skip the file header: fheader = zef_file.read(sizeFileHeader) @@ -1410,68 +1414,69 @@ zinfo.file_size = st.st_size zinfo.flag_bits = 0x00 - self.fp.seek(self.start_dir, 0) - zinfo.header_offset = self.fp.tell() # Start of header bytes - if zinfo.compress_type == ZIP_LZMA: - # Compressed data includes an end-of-stream (EOS) marker - zinfo.flag_bits |= 0x02 + with self._lock: + self.fp.seek(self.start_dir, 0) + zinfo.header_offset = self.fp.tell() # Start of header bytes + if zinfo.compress_type == ZIP_LZMA: + # Compressed data includes an end-of-stream (EOS) marker + zinfo.flag_bits |= 0x02 - self._writecheck(zinfo) - self._didModify = True + self._writecheck(zinfo) + self._didModify = True - if isdir: - zinfo.file_size = 0 - zinfo.compress_size = 0 - zinfo.CRC = 0 - zinfo.external_attr |= 0x10 # MS-DOS directory flag + if isdir: + zinfo.file_size = 0 + zinfo.compress_size = 0 + zinfo.CRC = 0 + zinfo.external_attr |= 0x10 # MS-DOS directory flag + self.filelist.append(zinfo) + self.NameToInfo[zinfo.filename] = zinfo + self.fp.write(zinfo.FileHeader(False)) + self.start_dir = self.fp.tell() + return + + cmpr = _get_compressor(zinfo.compress_type) + with open(filename, "rb") as fp: + # Must overwrite CRC and sizes with correct data later + zinfo.CRC = CRC = 0 + zinfo.compress_size = compress_size = 0 + # Compressed size can be larger than uncompressed size + zip64 = self._allowZip64 and \ + zinfo.file_size * 1.05 > ZIP64_LIMIT + self.fp.write(zinfo.FileHeader(zip64)) + file_size = 0 + while 1: + buf = fp.read(1024 * 8) + if not buf: + break + file_size = file_size + len(buf) + CRC = crc32(buf, CRC) & 0xffffffff + if cmpr: + buf = cmpr.compress(buf) + compress_size = compress_size + len(buf) + self.fp.write(buf) + if cmpr: + buf = cmpr.flush() + compress_size = compress_size + len(buf) + self.fp.write(buf) + zinfo.compress_size = compress_size + else: + zinfo.compress_size = file_size + zinfo.CRC = CRC + zinfo.file_size = file_size + if not zip64 and self._allowZip64: + if file_size > ZIP64_LIMIT: + raise RuntimeError('File size has increased during compressing') + if compress_size > ZIP64_LIMIT: + raise RuntimeError('Compressed size larger than uncompressed size') + # Seek backwards and write file header (which will now include + # correct CRC and file sizes) + self.start_dir = self.fp.tell() # Preserve current position in file + self.fp.seek(zinfo.header_offset, 0) + self.fp.write(zinfo.FileHeader(zip64)) + self.fp.seek(self.start_dir, 0) self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo - self.fp.write(zinfo.FileHeader(False)) - self.start_dir = self.fp.tell() - return - - cmpr = _get_compressor(zinfo.compress_type) - with open(filename, "rb") as fp: - # Must overwrite CRC and sizes with correct data later - zinfo.CRC = CRC = 0 - zinfo.compress_size = compress_size = 0 - # Compressed size can be larger than uncompressed size - zip64 = self._allowZip64 and \ - zinfo.file_size * 1.05 > ZIP64_LIMIT - self.fp.write(zinfo.FileHeader(zip64)) - file_size = 0 - while 1: - buf = fp.read(1024 * 8) - if not buf: - break - file_size = file_size + len(buf) - CRC = crc32(buf, CRC) & 0xffffffff - if cmpr: - buf = cmpr.compress(buf) - compress_size = compress_size + len(buf) - self.fp.write(buf) - if cmpr: - buf = cmpr.flush() - compress_size = compress_size + len(buf) - self.fp.write(buf) - zinfo.compress_size = compress_size - else: - zinfo.compress_size = file_size - zinfo.CRC = CRC - zinfo.file_size = file_size - if not zip64 and self._allowZip64: - if file_size > ZIP64_LIMIT: - raise RuntimeError('File size has increased during compressing') - if compress_size > ZIP64_LIMIT: - raise RuntimeError('Compressed size larger than uncompressed size') - # Seek backwards and write file header (which will now include - # correct CRC and file sizes) - self.start_dir = self.fp.tell() # Preserve current position in file - self.fp.seek(zinfo.header_offset, 0) - self.fp.write(zinfo.FileHeader(zip64)) - self.fp.seek(self.start_dir, 0) - self.filelist.append(zinfo) - self.NameToInfo[zinfo.filename] = zinfo def writestr(self, zinfo_or_arcname, data, compress_type=None): """Write a file into the archive. The contents is 'data', which @@ -1498,38 +1503,39 @@ "Attempt to write to ZIP archive that was already closed") zinfo.file_size = len(data) # Uncompressed size - self.fp.seek(self.start_dir, 0) - zinfo.header_offset = self.fp.tell() # Start of header data - if compress_type is not None: - zinfo.compress_type = compress_type - if zinfo.compress_type == ZIP_LZMA: - # Compressed data includes an end-of-stream (EOS) marker - zinfo.flag_bits |= 0x02 + with self._lock: + self.fp.seek(self.start_dir, 0) + zinfo.header_offset = self.fp.tell() # Start of header data + if compress_type is not None: + zinfo.compress_type = compress_type + if zinfo.compress_type == ZIP_LZMA: + # Compressed data includes an end-of-stream (EOS) marker + zinfo.flag_bits |= 0x02 - self._writecheck(zinfo) - self._didModify = True - zinfo.CRC = crc32(data) & 0xffffffff # CRC-32 checksum - co = _get_compressor(zinfo.compress_type) - if co: - data = co.compress(data) + co.flush() - zinfo.compress_size = len(data) # Compressed size - else: - zinfo.compress_size = zinfo.file_size - zip64 = zinfo.file_size > ZIP64_LIMIT or \ - zinfo.compress_size > ZIP64_LIMIT - if zip64 and not self._allowZip64: - raise LargeZipFile("Filesize would require ZIP64 extensions") - self.fp.write(zinfo.FileHeader(zip64)) - self.fp.write(data) - if zinfo.flag_bits & 0x08: - # Write CRC and file sizes after the file data - fmt = ' ZIP64_LIMIT or \ + zinfo.compress_size > ZIP64_LIMIT + if zip64 and not self._allowZip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + self.fp.write(zinfo.FileHeader(zip64)) + self.fp.write(data) + if zinfo.flag_bits & 0x08: + # Write CRC and file sizes after the file data + fmt = ' ZIP64_LIMIT \ - or zinfo.compress_size > ZIP64_LIMIT: - extra.append(zinfo.file_size) - extra.append(zinfo.compress_size) - file_size = 0xffffffff - compress_size = 0xffffffff - else: - file_size = zinfo.file_size - compress_size = zinfo.compress_size - - if zinfo.header_offset > ZIP64_LIMIT: - extra.append(zinfo.header_offset) - header_offset = 0xffffffff - else: - header_offset = zinfo.header_offset - - extra_data = zinfo.extra - min_version = 0 - if extra: - # Append a ZIP64 field to the extra's - extra_data = struct.pack( - ' ZIP_FILECOUNT_LIMIT: - requires_zip64 = "Files count" - elif centDirOffset > ZIP64_LIMIT: - requires_zip64 = "Central directory offset" - elif centDirSize > ZIP64_LIMIT: - requires_zip64 = "Central directory size" - if requires_zip64: - # Need to write the ZIP64 end-of-archive records - if not self._allowZip64: - raise LargeZipFile(requires_zip64 + - " would require ZIP64 extensions") - zip64endrec = struct.pack( - structEndArchive64, stringEndArchive64, - 44, 45, 45, 0, 0, centDirCount, centDirCount, - centDirSize, centDirOffset) - self.fp.write(zip64endrec) - - zip64locrec = struct.pack( - structEndArchive64Locator, - stringEndArchive64Locator, 0, pos2, 1) - self.fp.write(zip64locrec) - centDirCount = min(centDirCount, 0xFFFF) - centDirSize = min(centDirSize, 0xFFFFFFFF) - centDirOffset = min(centDirOffset, 0xFFFFFFFF) - - endrec = struct.pack(structEndArchive, stringEndArchive, - 0, 0, centDirCount, centDirCount, - centDirSize, centDirOffset, len(self._comment)) - self.fp.write(endrec) - self.fp.write(self._comment) - self.fp.flush() + with self._lock: + self.fp.seek(self.start_dir, 0) + self._write_end_record() finally: fp = self.fp self.fp = None self._fpclose(fp) + def _write_end_record(self): + self.fp.seek(self.start_dir, 0) + for zinfo in self.filelist: # write central directory + dt = zinfo.date_time + dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] + dostime = dt[3] << 11 | dt[4] << 5 | (dt[5] // 2) + extra = [] + if zinfo.file_size > ZIP64_LIMIT \ + or zinfo.compress_size > ZIP64_LIMIT: + extra.append(zinfo.file_size) + extra.append(zinfo.compress_size) + file_size = 0xffffffff + compress_size = 0xffffffff + else: + file_size = zinfo.file_size + compress_size = zinfo.compress_size + + if zinfo.header_offset > ZIP64_LIMIT: + extra.append(zinfo.header_offset) + header_offset = 0xffffffff + else: + header_offset = zinfo.header_offset + + extra_data = zinfo.extra + min_version = 0 + if extra: + # Append a ZIP64 field to the extra's + extra_data = struct.pack( + ' ZIP_FILECOUNT_LIMIT: + requires_zip64 = "Files count" + elif centDirOffset > ZIP64_LIMIT: + requires_zip64 = "Central directory offset" + elif centDirSize > ZIP64_LIMIT: + requires_zip64 = "Central directory size" + if requires_zip64: + # Need to write the ZIP64 end-of-archive records + if not self._allowZip64: + raise LargeZipFile(requires_zip64 + + " would require ZIP64 extensions") + zip64endrec = struct.pack( + structEndArchive64, stringEndArchive64, + 44, 45, 45, 0, 0, centDirCount, centDirCount, + centDirSize, centDirOffset) + self.fp.write(zip64endrec) + + zip64locrec = struct.pack( + structEndArchive64Locator, + stringEndArchive64Locator, 0, pos2, 1) + self.fp.write(zip64locrec) + centDirCount = min(centDirCount, 0xFFFF) + centDirSize = min(centDirSize, 0xFFFFFFFF) + centDirOffset = min(centDirOffset, 0xFFFFFFFF) + + endrec = struct.pack(structEndArchive, stringEndArchive, + 0, 0, centDirCount, centDirCount, + centDirSize, centDirOffset, len(self._comment)) + self.fp.write(endrec) + self.fp.write(self._comment) + self.fp.flush() + def _fpclose(self, fp): assert self._fileRefCnt > 0 self._fileRefCnt -= 1 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -218,6 +218,9 @@ Library ------- +- Issue #14099: Writing to ZipFile and reading multiple ZipExtFiles is + threadsafe now. + - Issue #19361: JSON decoder now raises JSONDecodeError instead of ValueError. - Issue #18518: timeit now rejects statements which can't be compiled outside -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 15:05:19 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 14:05:19 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjkz?= =?utf-8?q?=2C_asyncio=3A_Rewrite_IocpProactor=2Econnect=5Fpipe=28=29_as_a?= =?utf-8?q?_coroutine?= Message-ID: <20150126140501.118210.94597@psf.io> https://hg.python.org/cpython/rev/99c3e304a4ea changeset: 94318:99c3e304a4ea branch: 3.4 user: Victor Stinner date: Mon Jan 26 15:04:03 2015 +0100 summary: Issue #23293, asyncio: Rewrite IocpProactor.connect_pipe() as a coroutine Use a coroutine with asyncio.sleep() instead of call_later() to ensure that the schedule call is cancelled. Add also a unit test cancelling connect_pipe(). files: Lib/asyncio/windows_events.py | 39 ++++----- Lib/test/test_asyncio/test_windows_events.py | 13 +++ 2 files changed, 31 insertions(+), 21 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -518,28 +518,25 @@ return self._register(ov, pipe, finish_accept_pipe) - def _connect_pipe(self, fut, address, delay): - # Unfortunately there is no way to do an overlapped connect to a pipe. - # Call CreateFile() in a loop until it doesn't fail with - # ERROR_PIPE_BUSY - try: - handle = _overlapped.ConnectPipe(address) - except OSError as exc: - if exc.winerror == _overlapped.ERROR_PIPE_BUSY: - # Polling: retry later - delay = min(delay * 2, CONNECT_PIPE_MAX_DELAY) - self._loop.call_later(delay, - self._connect_pipe, fut, address, delay) - else: - fut.set_exception(exc) - else: - pipe = windows_utils.PipeHandle(handle) - fut.set_result(pipe) + @coroutine + def connect_pipe(self, address): + delay = CONNECT_PIPE_INIT_DELAY + while True: + # Unfortunately there is no way to do an overlapped connect to a pipe. + # Call CreateFile() in a loop until it doesn't fail with + # ERROR_PIPE_BUSY + try: + handle = _overlapped.ConnectPipe(address) + break + except OSError as exc: + if exc.winerror != _overlapped.ERROR_PIPE_BUSY: + raise - def connect_pipe(self, address): - fut = futures.Future(loop=self._loop) - self._connect_pipe(fut, address, CONNECT_PIPE_INIT_DELAY) - return fut + # ConnectPipe() failed with ERROR_PIPE_BUSY: retry later + delay = min(delay * 2, CONNECT_PIPE_MAX_DELAY) + yield from tasks.sleep(delay, loop=self._loop) + + return windows_utils.PipeHandle(handle) def wait_for_handle(self, handle, timeout=None): """Wait for a handle. diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -1,6 +1,7 @@ import os import sys import unittest +from unittest import mock if sys.platform != 'win32': raise unittest.SkipTest('Windows only') @@ -91,6 +92,18 @@ return 'done' + def test_connect_pipe_cancel(self): + exc = OSError() + exc.winerror = _overlapped.ERROR_PIPE_BUSY + with mock.patch.object(_overlapped, 'ConnectPipe', side_effect=exc) as connect: + coro = self.loop._proactor.connect_pipe('pipe_address') + task = self.loop.create_task(coro) + + # check that it's possible to cancel connect_pipe() + task.cancel() + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(task) + def test_wait_for_handle(self): event = _overlapped.CreateEvent(None, True, False, None) self.addCleanup(_winapi.CloseHandle, event) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 15:05:20 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 14:05:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogUGlw?= =?utf-8?q?eHandle=2Efileno=28=29_now_raises_an_exception_if_the_pipe_is_c?= =?utf-8?q?losed?= Message-ID: <20150126140501.87129.23560@psf.io> https://hg.python.org/cpython/rev/7ade2d9c3b91 changeset: 94317:7ade2d9c3b91 branch: 3.4 user: Victor Stinner date: Mon Jan 26 15:03:44 2015 +0100 summary: asyncio: PipeHandle.fileno() now raises an exception if the pipe is closed files: Lib/asyncio/windows_utils.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -147,6 +147,8 @@ return self._handle def fileno(self): + if self._handle is None: + raise ValueError("I/O operatioon on closed pipe") return self._handle def close(self, *, CloseHandle=_winapi.CloseHandle): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 15:05:20 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 14:05:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?q?_ProactorEventLoop=2Estart=5Fserving=5Fpipe=28=29?= Message-ID: <20150126140501.75788.63607@psf.io> https://hg.python.org/cpython/rev/619f289f604b changeset: 94316:619f289f604b branch: 3.4 parent: 94312:680b47c96e08 user: Victor Stinner date: Mon Jan 26 15:03:20 2015 +0100 summary: asyncio: Fix ProactorEventLoop.start_serving_pipe() If a client connected before the server was closed: drop the client (close the pipe) and exit. files: Lib/asyncio/windows_events.py | 14 +++++++++++++- 1 files changed, 13 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -257,7 +257,7 @@ def _server_pipe_handle(self, first): # Return a wrapper for a new pipe handle. - if self._address is None: + if self.closed(): return None flags = _winapi.PIPE_ACCESS_DUPLEX | _winapi.FILE_FLAG_OVERLAPPED if first: @@ -273,6 +273,9 @@ self._free_instances.add(pipe) return pipe + def closed(self): + return (self._address is None) + def close(self): if self._accept_pipe_future is not None: self._accept_pipe_future.cancel() @@ -325,12 +328,21 @@ if f: pipe = f.result() server._free_instances.discard(pipe) + + if server.closed(): + # A client connected before the server was closed: + # drop the client (close the pipe) and exit + pipe.close() + return + protocol = protocol_factory() self._make_duplex_pipe_transport( pipe, protocol, extra={'addr': address}) + pipe = server._get_unconnected_pipe() if pipe is None: return + f = self._proactor.accept_pipe(pipe) except OSError as exc: if pipe and pipe.fileno() != -1: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 15:05:20 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 14:05:20 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150126140502.84283.63192@psf.io> https://hg.python.org/cpython/rev/c2ee628585bd changeset: 94319:c2ee628585bd parent: 94315:9cbf9f96920d parent: 94318:99c3e304a4ea user: Victor Stinner date: Mon Jan 26 15:04:15 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/windows_events.py | 53 +++++---- Lib/asyncio/windows_utils.py | 2 + Lib/test/test_asyncio/test_windows_events.py | 13 ++ 3 files changed, 46 insertions(+), 22 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -257,7 +257,7 @@ def _server_pipe_handle(self, first): # Return a wrapper for a new pipe handle. - if self._address is None: + if self.closed(): return None flags = _winapi.PIPE_ACCESS_DUPLEX | _winapi.FILE_FLAG_OVERLAPPED if first: @@ -273,6 +273,9 @@ self._free_instances.add(pipe) return pipe + def closed(self): + return (self._address is None) + def close(self): if self._accept_pipe_future is not None: self._accept_pipe_future.cancel() @@ -325,12 +328,21 @@ if f: pipe = f.result() server._free_instances.discard(pipe) + + if server.closed(): + # A client connected before the server was closed: + # drop the client (close the pipe) and exit + pipe.close() + return + protocol = protocol_factory() self._make_duplex_pipe_transport( pipe, protocol, extra={'addr': address}) + pipe = server._get_unconnected_pipe() if pipe is None: return + f = self._proactor.accept_pipe(pipe) except OSError as exc: if pipe and pipe.fileno() != -1: @@ -506,28 +518,25 @@ return self._register(ov, pipe, finish_accept_pipe) - def _connect_pipe(self, fut, address, delay): - # Unfortunately there is no way to do an overlapped connect to a pipe. - # Call CreateFile() in a loop until it doesn't fail with - # ERROR_PIPE_BUSY - try: - handle = _overlapped.ConnectPipe(address) - except OSError as exc: - if exc.winerror == _overlapped.ERROR_PIPE_BUSY: - # Polling: retry later - delay = min(delay * 2, CONNECT_PIPE_MAX_DELAY) - self._loop.call_later(delay, - self._connect_pipe, fut, address, delay) - else: - fut.set_exception(exc) - else: - pipe = windows_utils.PipeHandle(handle) - fut.set_result(pipe) + @coroutine + def connect_pipe(self, address): + delay = CONNECT_PIPE_INIT_DELAY + while True: + # Unfortunately there is no way to do an overlapped connect to a pipe. + # Call CreateFile() in a loop until it doesn't fail with + # ERROR_PIPE_BUSY + try: + handle = _overlapped.ConnectPipe(address) + break + except OSError as exc: + if exc.winerror != _overlapped.ERROR_PIPE_BUSY: + raise - def connect_pipe(self, address): - fut = futures.Future(loop=self._loop) - self._connect_pipe(fut, address, CONNECT_PIPE_INIT_DELAY) - return fut + # ConnectPipe() failed with ERROR_PIPE_BUSY: retry later + delay = min(delay * 2, CONNECT_PIPE_MAX_DELAY) + yield from tasks.sleep(delay, loop=self._loop) + + return windows_utils.PipeHandle(handle) def wait_for_handle(self, handle, timeout=None): """Wait for a handle. diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -147,6 +147,8 @@ return self._handle def fileno(self): + if self._handle is None: + raise ValueError("I/O operatioon on closed pipe") return self._handle def close(self, *, CloseHandle=_winapi.CloseHandle): diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -1,6 +1,7 @@ import os import sys import unittest +from unittest import mock if sys.platform != 'win32': raise unittest.SkipTest('Windows only') @@ -91,6 +92,18 @@ return 'done' + def test_connect_pipe_cancel(self): + exc = OSError() + exc.winerror = _overlapped.ERROR_PIPE_BUSY + with mock.patch.object(_overlapped, 'ConnectPipe', side_effect=exc) as connect: + coro = self.loop._proactor.connect_pipe('pipe_address') + task = self.loop.create_task(coro) + + # check that it's possible to cancel connect_pipe() + task.cancel() + with self.assertRaises(asyncio.CancelledError): + self.loop.run_until_complete(task) + def test_wait_for_handle(self): event = _overlapped.CreateEvent(None, True, False, None) self.addCleanup(_winapi.CloseHandle, event) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 15:24:46 2015 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 26 Jan 2015 14:24:46 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_ensure_ilen_is_initialized?= =?utf-8?q?_when_it_is_assigned_to_len?= Message-ID: <20150126142347.84275.25152@psf.io> https://hg.python.org/cpython/rev/1c0dc8daa280 changeset: 94320:1c0dc8daa280 user: Benjamin Peterson date: Mon Jan 26 09:23:41 2015 -0500 summary: ensure ilen is initialized when it is assigned to len files: Objects/bytesobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -870,9 +870,9 @@ temp = format_long(iobj, flags, prec, c, &pbuf, &ilen); Py_DECREF(iobj); - len = ilen; if (!temp) goto error; + len = ilen; sign = 1; } else { -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 17:05:30 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 16:05:30 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320284=3A_Fix_a_co?= =?utf-8?q?mpilation_warning_on_Windows?= Message-ID: <20150126154456.55098.84248@psf.io> https://hg.python.org/cpython/rev/db7ec64aac39 changeset: 94322:db7ec64aac39 user: Victor Stinner date: Mon Jan 26 16:43:36 2015 +0100 summary: Issue #20284: Fix a compilation warning on Windows Explicitly cast the long to char. files: Objects/bytesobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -529,7 +529,7 @@ "%c requires an integer in range(256) or a single byte"); goto error; } - buf[0] = ival; + buf[0] = (char)ival; } Py_XDECREF(w); buf[1] = '\0'; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 17:05:30 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 16:05:30 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315859=3A_PyUnicod?= =?utf-8?q?e=5FEncodeFSDefault=28=29=2C_PyUnicode=5FEncodeMBCS=28=29_and?= Message-ID: <20150126154456.55118.29717@psf.io> https://hg.python.org/cpython/rev/e124aab5d9a0 changeset: 94321:e124aab5d9a0 user: Victor Stinner date: Mon Jan 26 16:41:32 2015 +0100 summary: Issue #15859: PyUnicode_EncodeFSDefault(), PyUnicode_EncodeMBCS() and PyUnicode_EncodeCodePage() now raise an exception if the object is not an Unicode object. For PyUnicode_EncodeFSDefault(), it was already the case on platforms other than Windows. Patch written by Campbell Barton. files: Misc/ACKS | 1 + Misc/NEWS | 5 +++++ Objects/unicodeobject.c | 9 +++++---- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -89,6 +89,7 @@ Cesar Eduardo Barros Des Barry Ulf Bartelt +Campbell Barton Don Bashford Pior Bastida Nick Bastin diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,11 @@ Core and Builtins ----------------- +- Issue #15859: PyUnicode_EncodeFSDefault(), PyUnicode_EncodeMBCS() and + PyUnicode_EncodeCodePage() now raise an exception if the object is not an + Unicode object. For PyUnicode_EncodeFSDefault(), it was already the case on + platforms other than Windows. Patch written by Campbell Barton. + - Issue #21408: The default __ne__() now returns NotImplemented if __eq__() returned NotImplemented. Original patch by Martin Panter. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -7431,6 +7431,11 @@ Py_ssize_t offset; int chunk_len, ret, done; + if (!PyUnicode_Check(unicode)) { + PyErr_BadArgument(); + return NULL; + } + if (PyUnicode_READY(unicode) == -1) return NULL; len = PyUnicode_GET_LENGTH(unicode); @@ -7504,10 +7509,6 @@ PyObject * PyUnicode_AsMBCSString(PyObject *unicode) { - if (!PyUnicode_Check(unicode)) { - PyErr_BadArgument(); - return NULL; - } return PyUnicode_EncodeCodePage(CP_ACP, unicode, NULL); } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 17:05:30 2015 From: python-checkins at python.org (chris.angelico) Date: Mon, 26 Jan 2015 16:05:30 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Fix_percentage_markers_in_PEP?= =?utf-8?q?_485_per_Chris_Barker=27s_edits?= Message-ID: <20150126160503.118206.46746@psf.io> https://hg.python.org/peps/rev/87a26df775d2 changeset: 5680:87a26df775d2 user: Chris Angelico date: Tue Jan 27 03:04:55 2015 +1100 summary: Fix percentage markers in PEP 485 per Chris Barker's edits files: pep-0485.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0485.txt b/pep-0485.txt --- a/pep-0485.txt +++ b/pep-0485.txt @@ -179,11 +179,11 @@ Example: -For the question: "Is the value of a within x% of b?", Using b to +For the question: "Is the value of a within 10% of b?", Using b to scale the percent error clearly defines the result. However, as this approach is not symmetric, a may be within 10% of b, -but b is not within x% of a. Consider the case:: +but b is not within 10% of a. Consider the case:: a = 9.0 b = 10.0 -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Mon Jan 26 21:12:42 2015 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 26 Jan 2015 20:12:42 +0000 Subject: [Python-checkins] =?utf-8?q?benchmarks=3A_Expand_default_suite_wh?= =?utf-8?q?ile_removing_useless_benchmarks_from_it?= Message-ID: <20150126201235.118212.29281@psf.io> https://hg.python.org/benchmarks/rev/1bd1437ea49b changeset: 219:1bd1437ea49b user: Antoine Pitrou date: Mon Jan 26 21:12:30 2015 +0100 summary: Expand default suite while removing useless benchmarks from it files: perf.py | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/perf.py b/perf.py --- a/perf.py +++ b/perf.py @@ -2245,8 +2245,10 @@ # If you update the default group, be sure to update the module docstring, too. # An "all" group which includes every benchmark perf.py knows about is generated # automatically. -BENCH_GROUPS = {"default": ["2to3", "django_v2", "nbody", "slowspitfire", - "slowpickle", "slowunpickle", "spambayes"], +BENCH_GROUPS = {"default": ["2to3", "django_v2", "nbody", "spambayes", + "tornado_http", "fastpickle", "fastunpickle", + "html5lib", "regex_v8", "etree", "json_dump_v2", + "json_load"], "startup": ["normal_startup", "startup_nosite", "bzr_startup", "hg_startup"], "regex": ["regex_v8", "regex_effbot", "regex_compile"], @@ -2258,7 +2260,7 @@ "etree": ["etree_generate", "etree_parse", "etree_iterparse", "etree_process"], "apps": ["2to3", "chameleon", "html5lib", "rietveld", - "spambayes"], + "spambayes", "tornado_http"], "calls": ["call_simple", "call_method", "call_method_slots", "call_method_unknown"], "math": ["float", "nbody", "pidigits"], -- Repository URL: https://hg.python.org/benchmarks From python-checkins at python.org Mon Jan 26 22:32:34 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 21:32:34 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDk1?= =?utf-8?q?=2C_asyncio=3A_Fix_=5FWaitHandleFuture=2Ecancel=28=29?= Message-ID: <20150126213214.55120.23063@psf.io> https://hg.python.org/cpython/rev/36e80c6599aa changeset: 94323:36e80c6599aa branch: 3.4 parent: 94318:99c3e304a4ea user: Victor Stinner date: Mon Jan 26 22:30:28 2015 +0100 summary: Issue #23095, asyncio: Fix _WaitHandleFuture.cancel() If UnregisterWaitEx() fais with ERROR_IO_PENDING, it doesn't mean that the wait is unregistered yet. We still have to wait until the wait is cancelled. files: Lib/asyncio/windows_events.py | 37 ++++++++++------------ 1 files changed, 17 insertions(+), 20 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -126,14 +126,12 @@ return self._registered = False + wait_handle = self._wait_handle + self._wait_handle = None try: - _overlapped.UnregisterWait(self._wait_handle) + _overlapped.UnregisterWait(wait_handle) except OSError as exc: - self._wait_handle = None - if exc.winerror == _overlapped.ERROR_IO_PENDING: - # ERROR_IO_PENDING is not an error, the wait was unregistered - self._unregister_wait_cb(None) - elif exc.winerror != _overlapped.ERROR_IO_PENDING: + if exc.winerror != _overlapped.ERROR_IO_PENDING: context = { 'message': 'Failed to unregister the wait handle', 'exception': exc, @@ -142,9 +140,10 @@ if self._source_traceback: context['source_traceback'] = self._source_traceback self._loop.call_exception_handler(context) - else: - self._wait_handle = None - self._unregister_wait_cb(None) + return + # ERROR_IO_PENDING means that the unregister is pending + + self._unregister_wait_cb(None) def cancel(self): self._unregister_wait() @@ -209,14 +208,12 @@ return self._registered = False + wait_handle = self._wait_handle + self._wait_handle = None try: - _overlapped.UnregisterWaitEx(self._wait_handle, self._event) + _overlapped.UnregisterWaitEx(wait_handle, self._event) except OSError as exc: - self._wait_handle = None - if exc.winerror == _overlapped.ERROR_IO_PENDING: - # ERROR_IO_PENDING is not an error, the wait was unregistered - self._unregister_wait_cb(None) - elif exc.winerror != _overlapped.ERROR_IO_PENDING: + if exc.winerror != _overlapped.ERROR_IO_PENDING: context = { 'message': 'Failed to unregister the wait handle', 'exception': exc, @@ -225,11 +222,11 @@ if self._source_traceback: context['source_traceback'] = self._source_traceback self._loop.call_exception_handler(context) - else: - self._wait_handle = None - self._event_fut = self._proactor._wait_cancel( - self._event, - self._unregister_wait_cb) + return + # ERROR_IO_PENDING is not an error, the wait was unregistered + + self._event_fut = self._proactor._wait_cancel(self._event, + self._unregister_wait_cb) class PipeServer(object): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 22:32:34 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 21:32:34 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?= =?utf-8?q?ip_issue_204=3A_Fix_IocpProactor=2Erecv=28=29?= Message-ID: <20150126213214.84273.12078@psf.io> https://hg.python.org/cpython/rev/9532923cbf2b changeset: 94324:9532923cbf2b branch: 3.4 user: Victor Stinner date: Mon Jan 26 22:30:49 2015 +0100 summary: asyncio, Tulip issue 204: Fix IocpProactor.recv() If ReadFile() fails with ERROR_BROKEN_PIPE, the operation is not pending: don't register the overlapped. I don't know if WSARecv() can fail with ERROR_BROKEN_PIPE. Since Overlapped.WSARecv() already handled ERROR_BROKEN_PIPE, let me guess that it has the same behaviour than ReadFile(). files: Lib/asyncio/windows_events.py | 20 +++++++++++++------- Modules/overlapped.c | 4 ++-- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -406,13 +406,21 @@ self._results = [] return tmp + def _result(self, value): + fut = futures.Future(loop=self._loop) + fut.set_result(value) + return fut + def recv(self, conn, nbytes, flags=0): self._register_with_iocp(conn) ov = _overlapped.Overlapped(NULL) - if isinstance(conn, socket.socket): - ov.WSARecv(conn.fileno(), nbytes, flags) - else: - ov.ReadFile(conn.fileno(), nbytes) + try: + if isinstance(conn, socket.socket): + ov.WSARecv(conn.fileno(), nbytes, flags) + else: + ov.ReadFile(conn.fileno(), nbytes) + except BrokenPipeError: + return self._result(b'') def finish_recv(trans, key, ov): try: @@ -505,9 +513,7 @@ # ConnectNamePipe() failed with ERROR_PIPE_CONNECTED which means # that the pipe is connected. There is no need to wait for the # completion of the connection. - f = futures.Future(loop=self._loop) - f.set_result(pipe) - return f + return self._result(pipe) def finish_accept_pipe(trans, key, ov): ov.getresult() diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -730,7 +730,7 @@ switch (err) { case ERROR_BROKEN_PIPE: mark_as_completed(&self->overlapped); - Py_RETURN_NONE; + return SetFromWindowsErr(err); case ERROR_SUCCESS: case ERROR_MORE_DATA: case ERROR_IO_PENDING: @@ -789,7 +789,7 @@ switch (err) { case ERROR_BROKEN_PIPE: mark_as_completed(&self->overlapped); - Py_RETURN_NONE; + return SetFromWindowsErr(err); case ERROR_SUCCESS: case ERROR_MORE_DATA: case ERROR_IO_PENDING: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 22:32:34 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 21:32:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150126213215.55108.67880@psf.io> https://hg.python.org/cpython/rev/c16c2c409776 changeset: 94325:c16c2c409776 parent: 94322:db7ec64aac39 parent: 94324:9532923cbf2b user: Victor Stinner date: Mon Jan 26 22:31:03 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/windows_events.py | 57 ++++++++++++---------- Modules/overlapped.c | 4 +- 2 files changed, 32 insertions(+), 29 deletions(-) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -126,14 +126,12 @@ return self._registered = False + wait_handle = self._wait_handle + self._wait_handle = None try: - _overlapped.UnregisterWait(self._wait_handle) + _overlapped.UnregisterWait(wait_handle) except OSError as exc: - self._wait_handle = None - if exc.winerror == _overlapped.ERROR_IO_PENDING: - # ERROR_IO_PENDING is not an error, the wait was unregistered - self._unregister_wait_cb(None) - elif exc.winerror != _overlapped.ERROR_IO_PENDING: + if exc.winerror != _overlapped.ERROR_IO_PENDING: context = { 'message': 'Failed to unregister the wait handle', 'exception': exc, @@ -142,9 +140,10 @@ if self._source_traceback: context['source_traceback'] = self._source_traceback self._loop.call_exception_handler(context) - else: - self._wait_handle = None - self._unregister_wait_cb(None) + return + # ERROR_IO_PENDING means that the unregister is pending + + self._unregister_wait_cb(None) def cancel(self): self._unregister_wait() @@ -209,14 +208,12 @@ return self._registered = False + wait_handle = self._wait_handle + self._wait_handle = None try: - _overlapped.UnregisterWaitEx(self._wait_handle, self._event) + _overlapped.UnregisterWaitEx(wait_handle, self._event) except OSError as exc: - self._wait_handle = None - if exc.winerror == _overlapped.ERROR_IO_PENDING: - # ERROR_IO_PENDING is not an error, the wait was unregistered - self._unregister_wait_cb(None) - elif exc.winerror != _overlapped.ERROR_IO_PENDING: + if exc.winerror != _overlapped.ERROR_IO_PENDING: context = { 'message': 'Failed to unregister the wait handle', 'exception': exc, @@ -225,11 +222,11 @@ if self._source_traceback: context['source_traceback'] = self._source_traceback self._loop.call_exception_handler(context) - else: - self._wait_handle = None - self._event_fut = self._proactor._wait_cancel( - self._event, - self._unregister_wait_cb) + return + # ERROR_IO_PENDING is not an error, the wait was unregistered + + self._event_fut = self._proactor._wait_cancel(self._event, + self._unregister_wait_cb) class PipeServer(object): @@ -409,13 +406,21 @@ self._results = [] return tmp + def _result(self, value): + fut = futures.Future(loop=self._loop) + fut.set_result(value) + return fut + def recv(self, conn, nbytes, flags=0): self._register_with_iocp(conn) ov = _overlapped.Overlapped(NULL) - if isinstance(conn, socket.socket): - ov.WSARecv(conn.fileno(), nbytes, flags) - else: - ov.ReadFile(conn.fileno(), nbytes) + try: + if isinstance(conn, socket.socket): + ov.WSARecv(conn.fileno(), nbytes, flags) + else: + ov.ReadFile(conn.fileno(), nbytes) + except BrokenPipeError: + return self._result(b'') def finish_recv(trans, key, ov): try: @@ -508,9 +513,7 @@ # ConnectNamePipe() failed with ERROR_PIPE_CONNECTED which means # that the pipe is connected. There is no need to wait for the # completion of the connection. - f = futures.Future(loop=self._loop) - f.set_result(pipe) - return f + return self._result(pipe) def finish_accept_pipe(trans, key, ov): ov.getresult() diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -730,7 +730,7 @@ switch (err) { case ERROR_BROKEN_PIPE: mark_as_completed(&self->overlapped); - Py_RETURN_NONE; + return SetFromWindowsErr(err); case ERROR_SUCCESS: case ERROR_MORE_DATA: case ERROR_IO_PENDING: @@ -789,7 +789,7 @@ switch (err) { case ERROR_BROKEN_PIPE: mark_as_completed(&self->overlapped); - Py_RETURN_NONE; + return SetFromWindowsErr(err); case ERROR_SUCCESS: case ERROR_MORE_DATA: case ERROR_IO_PENDING: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 22:44:11 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 21:44:11 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150126214359.55114.37311@psf.io> https://hg.python.org/cpython/rev/233b43d6f333 changeset: 94327:233b43d6f333 parent: 94325:c16c2c409776 parent: 94326:0024942193e5 user: Victor Stinner date: Mon Jan 26 22:43:52 2015 +0100 summary: Merge 3.4 (asyncio) files: Modules/overlapped.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -1146,10 +1146,13 @@ if (Address == NULL) return NULL; + Py_BEGIN_ALLOW_THREADS PipeHandle = CreateFileW(Address, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL); + Py_END_ALLOW_THREADS + PyMem_Free(Address); if (PipeHandle == INVALID_HANDLE_VALUE) return SetFromWindowsErr(0); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 22:44:11 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 21:44:11 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgX292?= =?utf-8?q?erlapped=2EConnectPipe=28=29=3A_release_the_GIL?= Message-ID: <20150126214359.75800.69170@psf.io> https://hg.python.org/cpython/rev/0024942193e5 changeset: 94326:0024942193e5 branch: 3.4 parent: 94324:9532923cbf2b user: Victor Stinner date: Mon Jan 26 22:43:39 2015 +0100 summary: asyncio, _overlapped.ConnectPipe(): release the GIL files: Modules/overlapped.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Modules/overlapped.c b/Modules/overlapped.c --- a/Modules/overlapped.c +++ b/Modules/overlapped.c @@ -1146,10 +1146,13 @@ if (Address == NULL) return NULL; + Py_BEGIN_ALLOW_THREADS PipeHandle = CreateFileW(Address, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL); + Py_END_ALLOW_THREADS + PyMem_Free(Address); if (PipeHandle == INVALID_HANDLE_VALUE) return SetFromWindowsErr(0); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Jan 26 23:26:55 2015 From: python-checkins at python.org (victor.stinner) Date: Mon, 26 Jan 2015 22:26:55 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322286=2C_=2323321?= =?utf-8?q?=3A_Fix_failing_test_on_Windows_code_page_932?= Message-ID: <20150126222618.29635.7443@psf.io> https://hg.python.org/cpython/rev/1e8937861ee3 changeset: 94328:1e8937861ee3 user: Victor Stinner date: Mon Jan 26 23:26:11 2015 +0100 summary: Issue #22286, #23321: Fix failing test on Windows code page 932 There was a bug which was fixed. The unit test was also wrong. files: Lib/test/test_codecs.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2954,7 +2954,7 @@ (b'\x81\x00abc', 'strict', None), (b'\x81\x00abc', 'ignore', '\x00abc'), (b'\x81\x00abc', 'replace', '\ufffd\x00abc'), - (b'\x81\x00abc', 'backslashreplace', '\\xff\x00abc'), + (b'\x81\x00abc', 'backslashreplace', '\\x81\x00abc'), )) def test_cp1252(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 01:52:30 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 27 Jan 2015 00:52:30 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjg2?= =?utf-8?q?=3A_Fix_typo_in_the_tutorial=2E?= Message-ID: <20150127005227.29645.92584@psf.io> https://hg.python.org/cpython/rev/b3f0d7f50544 changeset: 94329:b3f0d7f50544 branch: 3.4 parent: 94326:0024942193e5 user: Berker Peksag date: Tue Jan 27 02:52:14 2015 +0200 summary: Issue #23286: Fix typo in the tutorial. Patch by Mayank Tripathi. files: Doc/tutorial/introduction.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -391,7 +391,7 @@ >>> squares[:] [1, 4, 9, 16, 25] -Lists also supports operations like concatenation:: +Lists also support operations like concatenation:: >>> squares + [36, 49, 64, 81, 100] [1, 4, 9, 16, 25, 36, 49, 64, 81, 100] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 01:52:30 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 27 Jan 2015 00:52:30 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2323286=3A_Fix_typo_in_the_tutorial=2E?= Message-ID: <20150127005227.29586.2336@psf.io> https://hg.python.org/cpython/rev/8cb151ff1575 changeset: 94330:8cb151ff1575 parent: 94328:1e8937861ee3 parent: 94329:b3f0d7f50544 user: Berker Peksag date: Tue Jan 27 02:52:40 2015 +0200 summary: Issue #23286: Fix typo in the tutorial. Patch by Mayank Tripathi. files: Doc/tutorial/introduction.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -391,7 +391,7 @@ >>> squares[:] [1, 4, 9, 16, 25] -Lists also supports operations like concatenation:: +Lists also support operations like concatenation:: >>> squares + [36, 49, 64, 81, 100] [1, 4, 9, 16, 25, 36, 49, 64, 81, 100] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 01:58:55 2015 From: python-checkins at python.org (berker.peksag) Date: Tue, 27 Jan 2015 00:58:55 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_whatsnew_entry_for_iss?= =?utf-8?b?dWUgIzUzMDku?= Message-ID: <20150127005853.74291.11153@psf.io> https://hg.python.org/cpython/rev/107669985805 changeset: 94331:107669985805 user: Berker Peksag date: Tue Jan 27 02:59:09 2015 +0200 summary: Add whatsnew entry for issue #5309. files: Doc/whatsnew/3.5.rst | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -159,6 +159,13 @@ don't provide any options to redirect it. (Contributed by Berker Peksag in :issue:`22389`.) +distutils +--------- + +* The ``build`` and ``build_ext`` commands now accept a ``-j`` + option to enable parallel building of extension modules. + (Contributed by Antoine Pitrou in :issue:`5309`.) + doctest ------- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 06:33:58 2015 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 27 Jan 2015 05:33:58 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323269=3A__Tighten?= =?utf-8?q?_search=5Floop_in_set=5Finsert=5Fclean=28=29?= Message-ID: <20150127053355.74305.48201@psf.io> https://hg.python.org/cpython/rev/0b2a3d764e63 changeset: 94332:0b2a3d764e63 user: Raymond Hettinger date: Mon Jan 26 21:33:48 2015 -0800 summary: Issue #23269: Tighten search_loop in set_insert_clean() Instead of masking and shifting every loopup, move the wrap-around test outside of the inner-loop. files: Objects/setobject.c | 23 +++++++++++++++++------ 1 files changed, 17 insertions(+), 6 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -138,17 +138,28 @@ setentry *entry; size_t perturb = hash; size_t mask = (size_t)so->mask; - size_t i = (size_t)hash; + size_t i = (size_t)hash & mask; size_t j; while (1) { - for (j = 0 ; j <= LINEAR_PROBES ; j++) { - entry = &table[(i + j) & mask]; - if (entry->key == NULL) - goto found_null; + entry = &table[i]; + if (entry->key == NULL) + goto found_null; + if (i + LINEAR_PROBES <= mask) { + for (j = 1; j <= LINEAR_PROBES; j++) { + entry = &table[i + j]; + if (entry->key == NULL) + goto found_null; + } + } else { + for (j = 1; j <= LINEAR_PROBES; j++) { + entry = &table[(i + j) & mask]; + if (entry->key == NULL) + goto found_null; + } } perturb >>= PERTURB_SHIFT; - i = i * 5 + 1 + perturb; + i = (i * 5 + 1 + perturb) & mask; } found_null: entry->key = key; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 06:54:48 2015 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 27 Jan 2015 05:54:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_unneeded_dummy_test?= =?utf-8?q?_from_the_set_search_loop_=28when_the_hashes_match_we?= Message-ID: <20150127055442.126631.2670@psf.io> https://hg.python.org/cpython/rev/ed8cc03eed9c changeset: 94333:ed8cc03eed9c user: Raymond Hettinger date: Mon Jan 26 21:54:35 2015 -0800 summary: Remove unneeded dummy test from the set search loop (when the hashes match we know the key is not a dummy). files: Include/setobject.h | 5 ++++- Objects/setobject.c | 15 +++++++++++---- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/Include/setobject.h b/Include/setobject.h --- a/Include/setobject.h +++ b/Include/setobject.h @@ -14,7 +14,10 @@ 2. Active: key != NULL and key != dummy 3. Dummy: key == dummy -The hash field of Unused or Dummy slots have no meaning. +The hash field of Unused slots have no meaning. +The hash field of Dummny slots are set to -1 +meaning that dummy entries can be detected by +either entry->key==dummy or by entry->hash==-1. */ #define PySet_MINSIZE 8 diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -65,8 +65,10 @@ return entry; while (1) { - if (entry->hash == hash && entry->key != dummy) { /* dummy match unlikely */ + if (entry->hash == hash) { PyObject *startkey = entry->key; + /* startkey cannot be a dummy because the dummy hash field is -1 */ + assert(startkey != dummy); if (startkey == key) return entry; if (PyUnicode_CheckExact(startkey) @@ -83,15 +85,18 @@ if (cmp > 0) /* likely */ return entry; } - if (entry->key == dummy && freeslot == NULL) + if (entry->hash == -1 && freeslot == NULL) { + assert(entry->key == dummy); freeslot = entry; + } for (j = 1 ; j <= LINEAR_PROBES ; j++) { entry = &table[(i + j) & mask]; if (entry->key == NULL) goto found_null; - if (entry->hash == hash && entry->key != dummy) { + if (entry->hash == hash) { PyObject *startkey = entry->key; + assert(startkey != dummy); if (startkey == key) return entry; if (PyUnicode_CheckExact(startkey) @@ -108,8 +113,10 @@ if (cmp > 0) return entry; } - if (entry->key == dummy && freeslot == NULL) + if (entry->hash == -1 && freeslot == NULL) { + assert(entry->key == dummy); freeslot = entry; + } } perturb >>= PERTURB_SHIFT; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 07:07:05 2015 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 27 Jan 2015 06:07:05 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Revert_unintended_part_of_?= =?utf-8?q?the_commit_=28the_key=3D=3Ddummy_test_wasn=27t_supposed_to?= Message-ID: <20150127060658.29584.72523@psf.io> https://hg.python.org/cpython/rev/5ed1db00dc59 changeset: 94334:5ed1db00dc59 user: Raymond Hettinger date: Mon Jan 26 22:06:43 2015 -0800 summary: Revert unintended part of the commit (the key==dummy test wasn't supposed to change). files: Objects/setobject.c | 8 ++------ 1 files changed, 2 insertions(+), 6 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -85,10 +85,8 @@ if (cmp > 0) /* likely */ return entry; } - if (entry->hash == -1 && freeslot == NULL) { - assert(entry->key == dummy); + if (entry->key == dummy && freeslot == NULL) freeslot = entry; - } for (j = 1 ; j <= LINEAR_PROBES ; j++) { entry = &table[(i + j) & mask]; @@ -113,10 +111,8 @@ if (cmp > 0) return entry; } - if (entry->hash == -1 && freeslot == NULL) { - assert(entry->key == dummy); + if (entry->key == dummy && freeslot == NULL) freeslot = entry; - } } perturb >>= PERTURB_SHIFT; -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Tue Jan 27 09:07:25 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 27 Jan 2015 09:07:25 +0100 Subject: [Python-checkins] Daily reference leaks (107669985805): sum=3 Message-ID: results for 107669985805 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloguN5rgT', '-x'] From python-checkins at python.org Tue Jan 27 10:41:49 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 27 Jan 2015 09:41:49 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMTkx?= =?utf-8?q?=3A_fnmatch_functions_that_use_caching_are_now_threadsafe=2E?= Message-ID: <20150127094133.29655.3648@psf.io> https://hg.python.org/cpython/rev/fe12c34c39eb changeset: 94335:fe12c34c39eb branch: 2.7 parent: 94311:ae42c4576438 user: Serhiy Storchaka date: Tue Jan 27 11:40:51 2015 +0200 summary: Issue #23191: fnmatch functions that use caching are now threadsafe. files: Lib/fnmatch.py | 16 ++++++++++------ Misc/NEWS | 2 ++ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/Lib/fnmatch.py b/Lib/fnmatch.py --- a/Lib/fnmatch.py +++ b/Lib/fnmatch.py @@ -47,12 +47,14 @@ import os,posixpath result=[] pat=os.path.normcase(pat) - if not pat in _cache: + try: + re_pat = _cache[pat] + except KeyError: res = translate(pat) if len(_cache) >= _MAXCACHE: _cache.clear() - _cache[pat] = re.compile(res) - match=_cache[pat].match + _cache[pat] = re_pat = re.compile(res) + match = re_pat.match if os.path is posixpath: # normcase on posix is NOP. Optimize it away from the loop. for name in names: @@ -71,12 +73,14 @@ its arguments. """ - if not pat in _cache: + try: + re_pat = _cache[pat] + except KeyError: res = translate(pat) if len(_cache) >= _MAXCACHE: _cache.clear() - _cache[pat] = re.compile(res) - return _cache[pat].match(name) is not None + _cache[pat] = re_pat = re.compile(res) + return re_pat.match(name) is not None def translate(pat): """Translate a shell PATTERN to a regular expression. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Library ------- +- Issue #23191: fnmatch functions that use caching are now threadsafe. + - Issue #18518: timeit now rejects statements which can't be compiled outside a function or a loop (e.g. "return" or "break"). -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 17:10:34 2015 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 27 Jan 2015 16:10:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_disable_ALPN_o?= =?utf-8?q?n_LibreSSL=2C_which_has_a_large_version_number=2C_but_not_ALPN?= Message-ID: <20150127161030.74285.31459@psf.io> https://hg.python.org/cpython/rev/f7fd2776e80d changeset: 94337:f7fd2776e80d branch: 2.7 parent: 94335:fe12c34c39eb user: Benjamin Peterson date: Tue Jan 27 11:10:18 2015 -0500 summary: disable ALPN on LibreSSL, which has a large version number, but not ALPN support (closes #23329) files: Modules/_ssl.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -106,7 +106,7 @@ #endif /* ALPN added in OpenSSL 1.0.2 */ -#if OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) +#if !defined(LIBRESSL_VERSION_NUMBER) && OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) # define HAVE_ALPN #endif -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 17:10:35 2015 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 27 Jan 2015 16:10:35 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_disable_ALPN_on_LibreSSL?= =?utf-8?q?=2C_which_has_a_large_version_number=2C_but_not_ALPN?= Message-ID: <20150127161030.29653.19527@psf.io> https://hg.python.org/cpython/rev/53e94a687570 changeset: 94336:53e94a687570 parent: 94334:5ed1db00dc59 user: Benjamin Peterson date: Tue Jan 27 11:10:18 2015 -0500 summary: disable ALPN on LibreSSL, which has a large version number, but not ALPN support (closes #23329) files: Modules/_ssl.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -110,7 +110,7 @@ #endif /* ALPN added in OpenSSL 1.0.2 */ -#if OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) +#if !defined(LIBRESSL_VERSION_NUMBER) && OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) # define HAVE_ALPN #endif -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 21:33:37 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 27 Jan 2015 20:33:37 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <20150127203336.29582.32270@psf.io> https://hg.python.org/cpython/rev/1fef24e8a78b changeset: 94341:1fef24e8a78b branch: 2.7 parent: 94338:e6b9e277fbf4 parent: 94337:f7fd2776e80d user: Serhiy Storchaka date: Tue Jan 27 22:32:34 2015 +0200 summary: Merge heads files: Modules/_ssl.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -106,7 +106,7 @@ #endif /* ALPN added in OpenSSL 1.0.2 */ -#if OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) +#if !defined(LIBRESSL_VERSION_NUMBER) && OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined(OPENSSL_NO_TLSEXT) # define HAVE_ALPN #endif -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 21:33:37 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 27 Jan 2015 20:33:37 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzIzMDU1?= =?utf-8?q?=3A_Fixed_a_buffer_overflow_in_PyUnicode=5FFromFormatV=2E__Anal?= =?utf-8?q?ysis?= Message-ID: <20150127203335.29645.11431@psf.io> https://hg.python.org/cpython/rev/f849f937f78c changeset: 94339:f849f937f78c branch: 3.2 parent: 94018:1ce98e85929d user: Serhiy Storchaka date: Tue Jan 27 22:18:34 2015 +0200 summary: Issue #23055: Fixed a buffer overflow in PyUnicode_FromFormatV. Analysis and fix by Guido Vranken. files: Lib/test/test_unicode.py | 141 ++++++++++++++++++++++++-- Misc/NEWS | 12 ++ Objects/unicodeobject.c | 37 +++--- 3 files changed, 159 insertions(+), 31 deletions(-) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1661,7 +1661,10 @@ # Test PyUnicode_FromFormat() def test_from_format(self): support.import_module('ctypes') - from ctypes import pythonapi, py_object, c_int + from ctypes import ( + pythonapi, py_object, sizeof, + c_int, c_long, c_longlong, c_ssize_t, + c_uint, c_ulong, c_ulonglong, c_size_t, c_void_p) if sys.maxunicode == 65535: name = "PyUnicodeUCS2_FromFormat" else: @@ -1675,9 +1678,13 @@ for arg in args) return _PyUnicode_FromFormat(format, *cargs) + def check_format(expected, format, *args): + text = PyUnicode_FromFormat(format, *args) + self.assertEqual(expected, text) + # ascii format, non-ascii argument - text = PyUnicode_FromFormat(b'ascii\x7f=%U', 'unicode\xe9') - self.assertEqual(text, 'ascii\x7f=unicode\xe9') + check_format('ascii\x7f=unicode\xe9', + b'ascii\x7f=%U', 'unicode\xe9') # non-ascii format, ascii argument: ensure that PyUnicode_FromFormatV() # raises an error @@ -1686,25 +1693,131 @@ 'string, got a non-ASCII byte: 0xe9$', PyUnicode_FromFormat, b'unicode\xe9=%s', 'ascii') - self.assertEqual(PyUnicode_FromFormat(b'%c', c_int(0xabcd)), '\uabcd') - self.assertEqual(PyUnicode_FromFormat(b'%c', c_int(0x10ffff)), '\U0010ffff') + # test "%c" + check_format('\uabcd', + b'%c', c_int(0xabcd)) + check_format('\U0010ffff', + b'%c', c_int(0x10ffff)) + with self.assertRaises(OverflowError): + PyUnicode_FromFormat(b'%c', c_int(0x110000)) + # Issue #18183 + check_format('\U00010000\U00100000', + b'%c%c', c_int(0x10000), c_int(0x100000)) - # other tests - text = PyUnicode_FromFormat(b'%%A:%A', 'abc\xe9\uabcd\U0010ffff') - self.assertEqual(text, r"%A:'abc\xe9\uabcd\U0010ffff'") + # test "%" + check_format('%', + b'%') + check_format('%', + b'%%') + check_format('%s', + b'%%s') + check_format('[%]', + b'[%%]') + check_format('%abc', + b'%%%s', b'abc') - text = PyUnicode_FromFormat(b'repr=%V', 'abc', b'xyz') - self.assertEqual(text, 'repr=abc') + # test %S + check_format("repr=\u20acABC", + b'repr=%S', '\u20acABC') + + # test %R + check_format("repr='\u20acABC'", + b'repr=%R', '\u20acABC') + + # test integer formats (%i, %d, %u) + check_format('010', + b'%03i', c_int(10)) + check_format('0010', + b'%0.4i', c_int(10)) + check_format('-123', + b'%i', c_int(-123)) + + check_format('-123', + b'%d', c_int(-123)) + check_format('-123', + b'%ld', c_long(-123)) + check_format('-123', + b'%lld', c_longlong(-123)) + check_format('-123', + b'%zd', c_ssize_t(-123)) + + check_format('123', + b'%u', c_uint(123)) + check_format('123', + b'%lu', c_ulong(123)) + check_format('123', + b'%llu', c_ulonglong(123)) + check_format('123', + b'%zu', c_size_t(123)) + + # test long output + min_longlong = -(2 ** (8 * sizeof(c_longlong) - 1)) + max_longlong = -min_longlong - 1 + check_format(str(min_longlong), + b'%lld', c_longlong(min_longlong)) + check_format(str(max_longlong), + b'%lld', c_longlong(max_longlong)) + max_ulonglong = 2 ** (8 * sizeof(c_ulonglong)) - 1 + check_format(str(max_ulonglong), + b'%llu', c_ulonglong(max_ulonglong)) + PyUnicode_FromFormat(b'%p', c_void_p(-1)) + + # test padding (width and/or precision) + check_format('123'.rjust(10, '0'), + b'%010i', c_int(123)) + check_format('123'.rjust(100), + b'%100i', c_int(123)) + check_format('123'.rjust(100, '0'), + b'%.100i', c_int(123)) + check_format('123'.rjust(80, '0').rjust(100), + b'%100.80i', c_int(123)) + + check_format('123'.rjust(10, '0'), + b'%010u', c_uint(123)) + check_format('123'.rjust(100), + b'%100u', c_uint(123)) + check_format('123'.rjust(100, '0'), + b'%.100u', c_uint(123)) + check_format('123'.rjust(80, '0').rjust(100), + b'%100.80u', c_uint(123)) + + check_format('123'.rjust(10, '0'), + b'%010x', c_int(0x123)) + check_format('123'.rjust(100), + b'%100x', c_int(0x123)) + check_format('123'.rjust(100, '0'), + b'%.100x', c_int(0x123)) + check_format('123'.rjust(80, '0').rjust(100), + b'%100.80x', c_int(0x123)) + + # test %A + check_format(r"%A:'abc\xe9\uabcd\U0010ffff'", + b'%%A:%A', 'abc\xe9\uabcd\U0010ffff') + + # test %V + check_format('repr=abc', + b'repr=%V', 'abc', b'xyz') # Test string decode from parameter of %s using utf-8. # b'\xe4\xba\xba\xe6\xb0\x91' is utf-8 encoded byte sequence of # '\u4eba\u6c11' - text = PyUnicode_FromFormat(b'repr=%V', None, b'\xe4\xba\xba\xe6\xb0\x91') - self.assertEqual(text, 'repr=\u4eba\u6c11') + check_format('repr=\u4eba\u6c11', + b'repr=%V', None, b'\xe4\xba\xba\xe6\xb0\x91') #Test replace error handler. - text = PyUnicode_FromFormat(b'repr=%V', None, b'abc\xff') - self.assertEqual(text, 'repr=abc\ufffd') + check_format('repr=abc\ufffd', + b'repr=%V', None, b'abc\xff') + + # not supported: copy the raw format string. these tests are just here + # to check for crashs and should not be considered as specifications + check_format('%s', + b'%1%s', b'abc') + check_format('%1abc', + b'%1abc') + check_format('%+i', + b'%+i', c_int(10)) + check_format('%s', + b'%.%s', b'abc') # Test PyUnicode_AsWideChar() def test_aswidechar(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,18 @@ Python News +++++++++++ +What's New in Python 3.2.7? +============================ + +*Release date: XXXX-XX-XX* + +Core and Builtins +----------------- + +- Issue #23055: Fixed a buffer overflow in PyUnicode_FromFormatV. Analysis + and fix by Guido Vranken. + + What's New in Python 3.2.6? =========================== diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -759,15 +759,10 @@ * result in an array) */ for (f = format; *f; f++) { if (*f == '%') { - if (*(f+1)=='%') - continue; - if (*(f+1)=='S' || *(f+1)=='R' || *(f+1)=='A' || *(f+1) == 'V') - ++callcount; - while (Py_ISDIGIT((unsigned)*f)) - width = (width*10) + *f++ - '0'; - while (*++f && *f != '%' && !Py_ISALPHA((unsigned)*f)) - ; - if (*f == 's') + f++; + while (*f && *f != '%' && !Py_ISALPHA((unsigned)*f)) + f++; + if (*f == 's' || *f=='S' || *f=='R' || *f=='A' || *f=='V') ++callcount; } else if (128 <= (unsigned char)*f) { @@ -794,12 +789,16 @@ #ifdef HAVE_LONG_LONG int longlongflag = 0; #endif - const char* p = f; + const char* p = f++; width = 0; while (Py_ISDIGIT((unsigned)*f)) width = (width*10) + *f++ - '0'; - while (*++f && *f != '%' && !Py_ISALPHA((unsigned)*f)) - ; + precision = 0; + if (*f == '.') { + f++; + while (Py_ISDIGIT((unsigned)*f)) + precision = (precision*10) + *f++ - '0'; + } /* skip the 'l' or 'z' in {%ld, %zd, %lu, %zu} since * they don't affect the amount of space we reserve. @@ -823,16 +822,18 @@ switch (*f) { case 'c': { + int ordinal = va_arg(count, int); + if (ordinal < 0 || ordinal > 0x10ffff) { + PyErr_SetString(PyExc_OverflowError, + "%c arg not in range(0x110000)"); + goto fail; + } #ifndef Py_UNICODE_WIDE - int ordinal = va_arg(count, int); if (ordinal > 0xffff) n += 2; else - n++; -#else - (void)va_arg(count, int); +#endif n++; -#endif break; } case '%': @@ -840,6 +841,8 @@ break; case 'd': case 'u': case 'i': case 'x': (void) va_arg(count, int); + if (width < precision) + width = precision; #ifdef HAVE_LONG_LONG if (longlongflag) { if (width < MAX_LONG_LONG_CHARS) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 21:33:37 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 27 Jan 2015 20:33:37 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMDU1?= =?utf-8?q?=3A_Fixed_a_buffer_overflow_in_PyUnicode=5FFromFormatV=2E__Anal?= =?utf-8?q?ysis?= Message-ID: <20150127203335.130328.20326@psf.io> https://hg.python.org/cpython/rev/e6b9e277fbf4 changeset: 94338:e6b9e277fbf4 branch: 2.7 parent: 94335:fe12c34c39eb user: Serhiy Storchaka date: Tue Jan 27 22:17:56 2015 +0200 summary: Issue #23055: Fixed a buffer overflow in PyUnicode_FromFormatV. Analysis and fix by Guido Vranken. files: Lib/test/test_unicode.py | 40 ++++++++++++++++++++++++++++ Misc/NEWS | 3 ++ Objects/unicodeobject.c | 25 +++++++++-------- 3 files changed, 56 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1700,6 +1700,9 @@ if sys.maxunicode > 0xffff: check_format(u'\U0010ffff', b'%c', c_int(0x10ffff)) + else: + with self.assertRaises(OverflowError): + PyUnicode_FromFormat(b'%c', c_int(0x10000)) with self.assertRaises(OverflowError): PyUnicode_FromFormat(b'%c', c_int(0x110000)) # Issue #18183 @@ -1750,8 +1753,45 @@ b'%zu', c_size_t(123)) # test long output + min_long = -(2 ** (8 * sizeof(c_long) - 1)) + max_long = -min_long - 1 + check_format(unicode(min_long), + b'%ld', c_long(min_long)) + check_format(unicode(max_long), + b'%ld', c_long(max_long)) + max_ulong = 2 ** (8 * sizeof(c_ulong)) - 1 + check_format(unicode(max_ulong), + b'%lu', c_ulong(max_ulong)) PyUnicode_FromFormat(b'%p', c_void_p(-1)) + # test padding (width and/or precision) + check_format(u'123'.rjust(10, u'0'), + b'%010i', c_int(123)) + check_format(u'123'.rjust(100), + b'%100i', c_int(123)) + check_format(u'123'.rjust(100, u'0'), + b'%.100i', c_int(123)) + check_format(u'123'.rjust(80, u'0').rjust(100), + b'%100.80i', c_int(123)) + + check_format(u'123'.rjust(10, u'0'), + b'%010u', c_uint(123)) + check_format(u'123'.rjust(100), + b'%100u', c_uint(123)) + check_format(u'123'.rjust(100, u'0'), + b'%.100u', c_uint(123)) + check_format(u'123'.rjust(80, u'0').rjust(100), + b'%100.80u', c_uint(123)) + + check_format(u'123'.rjust(10, u'0'), + b'%010x', c_int(0x123)) + check_format(u'123'.rjust(100), + b'%100x', c_int(0x123)) + check_format(u'123'.rjust(100, u'0'), + b'%.100x', c_int(0x123)) + check_format(u'123'.rjust(80, u'0').rjust(100), + b'%100.80x', c_int(0x123)) + # test %V check_format(u'repr=abc', b'repr=%V', u'abc', b'xyz') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #23055: Fixed a buffer overflow in PyUnicode_FromFormatV. Analysis + and fix by Guido Vranken. + - Issue #23048: Fix jumping out of an infinite while loop in the pdb. Library diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -735,15 +735,10 @@ * objects once during step 3 and put the result in an array) */ for (f = format; *f; f++) { if (*f == '%') { - if (*(f+1)=='%') - continue; - if (*(f+1)=='S' || *(f+1)=='R') - ++callcount; - while (isdigit((unsigned)*f)) - width = (width*10) + *f++ - '0'; - while (*++f && *f != '%' && !isalpha((unsigned)*f)) - ; - if (*f == 's') + f++; + while (*f && *f != '%' && !isalpha((unsigned)*f)) + f++; + if (*f == 's' || *f=='S' || *f=='R') ++callcount; } } @@ -760,12 +755,16 @@ /* step 3: figure out how large a buffer we need */ for (f = format; *f; f++) { if (*f == '%') { - const char* p = f; + const char* p = f++; width = 0; while (isdigit((unsigned)*f)) width = (width*10) + *f++ - '0'; - while (*++f && *f != '%' && !isalpha((unsigned)*f)) - ; + precision = 0; + if (*f == '.') { + f++; + while (isdigit((unsigned)*f)) + precision = (precision*10) + *f++ - '0'; + } /* skip the 'l' or 'z' in {%ld, %zd, %lu, %zu} since * they don't affect the amount of space we reserve. @@ -800,6 +799,8 @@ break; case 'd': case 'u': case 'i': case 'x': (void) va_arg(count, int); + if (width < precision) + width = precision; /* 20 bytes is enough to hold a 64-bit integer. Decimal takes the most space. This isn't enough for octal. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 21:33:37 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 27 Jan 2015 20:33:37 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIzMDU1?= =?utf-8?q?=3A_Fixed_a_buffer_overflow_in_PyUnicode=5FFromFormatV=2E__Anal?= =?utf-8?q?ysis?= Message-ID: <20150127203336.126637.59997@psf.io> https://hg.python.org/cpython/rev/6caed177a028 changeset: 94340:6caed177a028 branch: 3.3 parent: 94149:94696e457461 user: Serhiy Storchaka date: Tue Jan 27 22:18:46 2015 +0200 summary: Issue #23055: Fixed a buffer overflow in PyUnicode_FromFormatV. Analysis and fix by Guido Vranken. files: Lib/test/test_unicode.py | 157 ++++++++++++++++++++------ Misc/NEWS | 12 ++ Objects/unicodeobject.c | 2 + 3 files changed, 131 insertions(+), 40 deletions(-) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2016,9 +2016,10 @@ # Test PyUnicode_FromFormat() def test_from_format(self): support.import_module('ctypes') - from ctypes import (pythonapi, py_object, + from ctypes import ( + pythonapi, py_object, sizeof, c_int, c_long, c_longlong, c_ssize_t, - c_uint, c_ulong, c_ulonglong, c_size_t) + c_uint, c_ulong, c_ulonglong, c_size_t, c_void_p) name = "PyUnicode_FromFormat" _PyUnicode_FromFormat = getattr(pythonapi, name) _PyUnicode_FromFormat.restype = py_object @@ -2029,9 +2030,13 @@ for arg in args) return _PyUnicode_FromFormat(format, *cargs) + def check_format(expected, format, *args): + text = PyUnicode_FromFormat(format, *args) + self.assertEqual(expected, text) + # ascii format, non-ascii argument - text = PyUnicode_FromFormat(b'ascii\x7f=%U', 'unicode\xe9') - self.assertEqual(text, 'ascii\x7f=unicode\xe9') + check_format('ascii\x7f=unicode\xe9', + b'ascii\x7f=%U', 'unicode\xe9') # non-ascii format, ascii argument: ensure that PyUnicode_FromFormatV() # raises an error @@ -2041,64 +2046,136 @@ PyUnicode_FromFormat, b'unicode\xe9=%s', 'ascii') # test "%c" - self.assertEqual(PyUnicode_FromFormat(b'%c', c_int(0xabcd)), '\uabcd') - self.assertEqual(PyUnicode_FromFormat(b'%c', c_int(0x10ffff)), '\U0010ffff') + check_format('\uabcd', + b'%c', c_int(0xabcd)) + check_format('\U0010ffff', + b'%c', c_int(0x10ffff)) with self.assertRaises(OverflowError): PyUnicode_FromFormat(b'%c', c_int(0x110000)) # Issue #18183 - self.assertEqual( - PyUnicode_FromFormat(b'%c%c', c_int(0x10000), c_int(0x100000)), - '\U00010000\U00100000') + check_format('\U00010000\U00100000', + b'%c%c', c_int(0x10000), c_int(0x100000)) # test "%" - self.assertEqual(PyUnicode_FromFormat(b'%'), '%') - self.assertEqual(PyUnicode_FromFormat(b'%%'), '%') - self.assertEqual(PyUnicode_FromFormat(b'%%s'), '%s') - self.assertEqual(PyUnicode_FromFormat(b'[%%]'), '[%]') - self.assertEqual(PyUnicode_FromFormat(b'%%%s', b'abc'), '%abc') + check_format('%', + b'%') + check_format('%', + b'%%') + check_format('%s', + b'%%s') + check_format('[%]', + b'[%%]') + check_format('%abc', + b'%%%s', b'abc') + + # test %S + check_format("repr=\u20acABC", + b'repr=%S', '\u20acABC') + + # test %R + check_format("repr='\u20acABC'", + b'repr=%R', '\u20acABC') # test integer formats (%i, %d, %u) - self.assertEqual(PyUnicode_FromFormat(b'%03i', c_int(10)), '010') - self.assertEqual(PyUnicode_FromFormat(b'%0.4i', c_int(10)), '0010') - self.assertEqual(PyUnicode_FromFormat(b'%i', c_int(-123)), '-123') - self.assertEqual(PyUnicode_FromFormat(b'%li', c_long(-123)), '-123') - self.assertEqual(PyUnicode_FromFormat(b'%lli', c_longlong(-123)), '-123') - self.assertEqual(PyUnicode_FromFormat(b'%zi', c_ssize_t(-123)), '-123') + check_format('010', + b'%03i', c_int(10)) + check_format('0010', + b'%0.4i', c_int(10)) + check_format('-123', + b'%i', c_int(-123)) + check_format('-123', + b'%li', c_long(-123)) + check_format('-123', + b'%lli', c_longlong(-123)) + check_format('-123', + b'%zi', c_ssize_t(-123)) - self.assertEqual(PyUnicode_FromFormat(b'%d', c_int(-123)), '-123') - self.assertEqual(PyUnicode_FromFormat(b'%ld', c_long(-123)), '-123') - self.assertEqual(PyUnicode_FromFormat(b'%lld', c_longlong(-123)), '-123') - self.assertEqual(PyUnicode_FromFormat(b'%zd', c_ssize_t(-123)), '-123') + check_format('-123', + b'%d', c_int(-123)) + check_format('-123', + b'%ld', c_long(-123)) + check_format('-123', + b'%lld', c_longlong(-123)) + check_format('-123', + b'%zd', c_ssize_t(-123)) - self.assertEqual(PyUnicode_FromFormat(b'%u', c_uint(123)), '123') - self.assertEqual(PyUnicode_FromFormat(b'%lu', c_ulong(123)), '123') - self.assertEqual(PyUnicode_FromFormat(b'%llu', c_ulonglong(123)), '123') - self.assertEqual(PyUnicode_FromFormat(b'%zu', c_size_t(123)), '123') + check_format('123', + b'%u', c_uint(123)) + check_format('123', + b'%lu', c_ulong(123)) + check_format('123', + b'%llu', c_ulonglong(123)) + check_format('123', + b'%zu', c_size_t(123)) + + # test long output + min_longlong = -(2 ** (8 * sizeof(c_longlong) - 1)) + max_longlong = -min_longlong - 1 + check_format(str(min_longlong), + b'%lld', c_longlong(min_longlong)) + check_format(str(max_longlong), + b'%lld', c_longlong(max_longlong)) + max_ulonglong = 2 ** (8 * sizeof(c_ulonglong)) - 1 + check_format(str(max_ulonglong), + b'%llu', c_ulonglong(max_ulonglong)) + PyUnicode_FromFormat(b'%p', c_void_p(-1)) + + # test padding (width and/or precision) + check_format('123'.rjust(10, '0'), + b'%010i', c_int(123)) + check_format('123'.rjust(100), + b'%100i', c_int(123)) + check_format('123'.rjust(300, '0'), + b'%.300i', c_int(123)) + check_format('123'.rjust(80, '0').rjust(100), + b'%100.80i', c_int(123)) + + check_format('123'.rjust(10, '0'), + b'%010u', c_uint(123)) + check_format('123'.rjust(100), + b'%100u', c_uint(123)) + check_format('123'.rjust(300, '0'), + b'%.300u', c_uint(123)) + check_format('123'.rjust(80, '0').rjust(100), + b'%100.80u', c_uint(123)) + + check_format('123'.rjust(10, '0'), + b'%010x', c_int(0x123)) + check_format('123'.rjust(100), + b'%100x', c_int(0x123)) + check_format('123'.rjust(300, '0'), + b'%.300x', c_int(0x123)) + check_format('123'.rjust(80, '0').rjust(100), + b'%100.80x', c_int(0x123)) # test %A - text = PyUnicode_FromFormat(b'%%A:%A', 'abc\xe9\uabcd\U0010ffff') - self.assertEqual(text, r"%A:'abc\xe9\uabcd\U0010ffff'") + check_format(r"%A:'abc\xe9\uabcd\U0010ffff'", + b'%%A:%A', 'abc\xe9\uabcd\U0010ffff') # test %V - text = PyUnicode_FromFormat(b'repr=%V', 'abc', b'xyz') - self.assertEqual(text, 'repr=abc') + check_format('repr=abc', + b'repr=%V', 'abc', b'xyz') # Test string decode from parameter of %s using utf-8. # b'\xe4\xba\xba\xe6\xb0\x91' is utf-8 encoded byte sequence of # '\u4eba\u6c11' - text = PyUnicode_FromFormat(b'repr=%V', None, b'\xe4\xba\xba\xe6\xb0\x91') - self.assertEqual(text, 'repr=\u4eba\u6c11') + check_format('repr=\u4eba\u6c11', + b'repr=%V', None, b'\xe4\xba\xba\xe6\xb0\x91') #Test replace error handler. - text = PyUnicode_FromFormat(b'repr=%V', None, b'abc\xff') - self.assertEqual(text, 'repr=abc\ufffd') + check_format('repr=abc\ufffd', + b'repr=%V', None, b'abc\xff') # not supported: copy the raw format string. these tests are just here # to check for crashs and should not be considered as specifications - self.assertEqual(PyUnicode_FromFormat(b'%1%s', b'abc'), '%s') - self.assertEqual(PyUnicode_FromFormat(b'%1abc'), '%1abc') - self.assertEqual(PyUnicode_FromFormat(b'%+i', c_int(10)), '%+i') - self.assertEqual(PyUnicode_FromFormat(b'%.%s', b'abc'), '%.%s') + check_format('%s', + b'%1%s', b'abc') + check_format('%1abc', + b'%1abc') + check_format('%+i', + b'%+i', c_int(10)) + check_format('%.%s', + b'%.%s', b'abc') # Test PyUnicode_AsWideChar() @support.cpython_only diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,18 @@ Python News +++++++++++ +What's New in Python 3.3.7? +============================ + +*Release date: XXXX-XX-XX* + +Core and Builtins +----------------- + +- Issue #23055: Fixed a buffer overflow in PyUnicode_FromFormatV. Analysis + and fix by Guido Vranken. + + What's New in Python 3.3.6? =========================== diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -2335,6 +2335,8 @@ f--; } } + if (width < precision) + width = precision; if (*f == '\0') { /* bogus format "%.1" => go backward, f points to "1" */ f--; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Jan 27 21:45:30 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 27 Jan 2015 20:45:30 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5OTQ5?= =?utf-8?q?=3A_The_test=5Fxpickle_test_now_tests_compatibility_with_instal?= =?utf-8?q?led?= Message-ID: <20150127204528.130324.58586@psf.io> https://hg.python.org/cpython/rev/94d8524086bd changeset: 94342:94d8524086bd branch: 2.7 user: Serhiy Storchaka date: Tue Jan 27 22:44:45 2015 +0200 summary: Issue #19949: The test_xpickle test now tests compatibility with installed Python 2.7 and reports skipped tests. Based on patch by Zachary Ware. files: Lib/test/test_xpickle.py | 117 ++++++++++++--------------- Misc/NEWS | 2 + 2 files changed, 54 insertions(+), 65 deletions(-) diff --git a/Lib/test/test_xpickle.py b/Lib/test/test_xpickle.py --- a/Lib/test/test_xpickle.py +++ b/Lib/test/test_xpickle.py @@ -56,7 +56,7 @@ # Ignore fast return cPickle.loads(buf) -def have_python_version(name): +def have_python_version(name, cache={}): """Check whether the given name is a valid Python binary and has test.test_support. @@ -68,7 +68,9 @@ Returns: True if the name is valid, False otherwise. """ - return os.system(name + " -c 'import test.test_support'") == 0 + if name not in cache: + cache[name] = os.system(name + ' -c "import test.test_support"') == 0 + return cache[name] class AbstractCompatTests(AbstractPickleTests): @@ -81,6 +83,9 @@ self.assertTrue(self.python) self.assertTrue(self.module) self.assertTrue(self.error) + test_support.requires("xpickle") + if not have_python_version(self.python): + self.skipTest('%s not available' % self.python) def send_to_worker(self, python, obj, proto): """Bounce a pickled object through another version of Python. @@ -119,14 +124,9 @@ # These tests are disabled because they require some special setup # on the worker that's hard to keep in sync. - def test_global_ext1(self): - pass - - def test_global_ext2(self): - pass - - def test_global_ext4(self): - pass + test_global_ext1 = None + test_global_ext2 = None + test_global_ext4 = None # This is a cut-down version of pickletester's test_float. Backwards # compatibility for the values in for_bin_protos was explicitly broken in @@ -151,48 +151,34 @@ self.assertEqual(value, got) # Backwards compatibility was explicitly broken in r67934 to fix a bug. - def test_unicode_high_plane(self): - pass + test_unicode_high_plane = None # This tests a fix that's in 2.7 only - def test_dynamic_class(self): - pass + test_dynamic_class = None - if test_support.have_unicode: - # This is a cut-down version of pickletester's test_unicode. Backwards - # compatibility was explicitly broken in r67934 to fix a bug. - def test_unicode(self): - endcases = [u'', u'<\\u>', u'<\\\u1234>', u'<\n>', u'<\\>'] - for proto in pickletester.protocols: - for u in endcases: - p = self.dumps(u, proto) - u2 = self.loads(p) - self.assertEqual(u2, u) - - -def run_compat_test(python_name): - return (test_support.is_resource_enabled("xpickle") and - have_python_version(python_name)) + # This is a cut-down version of pickletester's test_unicode. Backwards + # compatibility was explicitly broken in r67934 to fix a bug. + @unittest.skipUnless(test_support.have_unicode, 'no unicode support') + def test_unicode(self): + endcases = [u'', u'<\\u>', u'<\\%c>' % 0x1234, u'<\n>', u'<\\>'] + for proto in pickletester.protocols: + for u in endcases: + p = self.dumps(u, proto) + u2 = self.loads(p) + self.assertEqual(u2, u) # Test backwards compatibility with Python 2.4. -if not run_compat_test("python2.4"): - class CPicklePython24Compat(unittest.TestCase): - pass -else: - class CPicklePython24Compat(AbstractCompatTests): +class CPicklePython24Compat(AbstractCompatTests): - module = cPickle - python = "python2.4" - error = cPickle.BadPickleGet + module = cPickle + python = "python2.4" + error = cPickle.BadPickleGet - # Disable these tests for Python 2.4. Making them pass would require - # nontrivially monkeypatching the pickletester module in the worker. - def test_reduce_calls_base(self): - pass - - def test_reduce_ex_calls_base(self): - pass + # Disable these tests for Python 2.4. Making them pass would require + # nontrivially monkeypatching the pickletester module in the worker. + test_reduce_calls_base = None + test_reduce_ex_calls_base = None class PicklePython24Compat(CPicklePython24Compat): @@ -201,15 +187,11 @@ # Test backwards compatibility with Python 2.5. -if not run_compat_test("python2.5"): - class CPicklePython25Compat(unittest.TestCase): - pass -else: - class CPicklePython25Compat(AbstractCompatTests): +class CPicklePython25Compat(AbstractCompatTests): - module = cPickle - python = "python2.5" - error = cPickle.BadPickleGet + module = cPickle + python = "python2.5" + error = cPickle.BadPickleGet class PicklePython25Compat(CPicklePython25Compat): @@ -218,15 +200,11 @@ # Test backwards compatibility with Python 2.6. -if not run_compat_test("python2.6"): - class CPicklePython26Compat(unittest.TestCase): - pass -else: - class CPicklePython26Compat(AbstractCompatTests): +class CPicklePython26Compat(AbstractCompatTests): - module = cPickle - python = "python2.6" - error = cPickle.BadPickleGet + module = cPickle + python = "python2.6" + error = cPickle.BadPickleGet class PicklePython26Compat(CPicklePython26Compat): @@ -234,6 +212,18 @@ error = KeyError +class CPicklePython27Compat(AbstractCompatTests): + + module = cPickle + python = "python2.7" + error = cPickle.BadPickleGet + +class PicklePython27Compat(CPicklePython26Compat): + + module = pickle + error = KeyError + + def worker_main(in_stream, out_stream): message = cPickle.load(in_stream) protocol, obj = message @@ -241,20 +231,17 @@ def test_main(): - if not test_support.is_resource_enabled("xpickle"): - print >>sys.stderr, "test_xpickle -- skipping backwards compat tests." - print >>sys.stderr, "Use 'regrtest.py -u xpickle' to run them." - sys.stderr.flush() - test_support.run_unittest( DumpCPickle_LoadPickle, DumpPickle_LoadCPickle, CPicklePython24Compat, CPicklePython25Compat, CPicklePython26Compat, + CPicklePython27Compat, PicklePython24Compat, PicklePython25Compat, PicklePython26Compat, + PicklePython27Compat, ) if __name__ == "__main__": diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -86,6 +86,8 @@ Tests ----- +- Issue #19949: The test_xpickle test now + - Issue #11578: Backported test for the timeit module. - Issue #22943: bsddb tests are locale independend now. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 00:35:01 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 27 Jan 2015 23:35:01 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogc3lu?= =?utf-8?q?c_with_Tulip?= Message-ID: <20150127233454.130344.53989@psf.io> https://hg.python.org/cpython/rev/d61d1e73674f changeset: 94343:d61d1e73674f branch: 3.4 parent: 94329:b3f0d7f50544 user: Victor Stinner date: Wed Jan 28 00:30:40 2015 +0100 summary: asyncio: sync with Tulip * Remove unused SSLProtocol._closing attribute * test_sslproto: skip test if ssl module is missing * Python issue #23208: Don't use the traceback of the current handle if we already know the traceback of the source. The handle may be more revelant, but having 3 tracebacks (handle, source, exception) becomes more difficult to read. The handle may be preferred later but it requires more work to make this choice. files: Lib/asyncio/base_events.py | 3 ++- Lib/asyncio/sslproto.py | 1 - Lib/test/test_asyncio/test_sslproto.py | 5 +++++ 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -956,7 +956,8 @@ else: exc_info = False - if (self._current_handle is not None + if ('source_traceback' not in context + and self._current_handle is not None and self._current_handle._source_traceback): context['handle_traceback'] = self._current_handle._source_traceback diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -408,7 +408,6 @@ self._write_buffer_size = 0 self._waiter = waiter - self._closing = False self._loop = loop self._app_protocol = app_protocol self._app_transport = _SSLProtocolTransport(self._loop, diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -2,6 +2,10 @@ import unittest from unittest import mock +try: + import ssl +except ImportError: + ssl = None import asyncio from asyncio import sslproto @@ -14,6 +18,7 @@ self.loop = asyncio.new_event_loop() self.set_event_loop(self.loop) + @unittest.skipIf(ssl is None, 'No ssl module') def test_cancel_handshake(self): # Python issue #23197: cancelling an handshake must not raise an # exception or log an error, even if the handshake failed -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 00:35:01 2015 From: python-checkins at python.org (victor.stinner) Date: Tue, 27 Jan 2015 23:35:01 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150127233454.130332.84228@psf.io> https://hg.python.org/cpython/rev/3b920a778484 changeset: 94344:3b920a778484 parent: 94336:53e94a687570 parent: 94343:d61d1e73674f user: Victor Stinner date: Wed Jan 28 00:30:51 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 3 ++- Lib/asyncio/sslproto.py | 1 - Lib/test/test_asyncio/test_sslproto.py | 5 +++++ 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -956,7 +956,8 @@ else: exc_info = False - if (self._current_handle is not None + if ('source_traceback' not in context + and self._current_handle is not None and self._current_handle._source_traceback): context['handle_traceback'] = self._current_handle._source_traceback diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -408,7 +408,6 @@ self._write_buffer_size = 0 self._waiter = waiter - self._closing = False self._loop = loop self._app_protocol = app_protocol self._app_transport = _SSLProtocolTransport(self._loop, diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -2,6 +2,10 @@ import unittest from unittest import mock +try: + import ssl +except ImportError: + ssl = None import asyncio from asyncio import sslproto @@ -14,6 +18,7 @@ self.loop = asyncio.new_event_loop() self.set_event_loop(self.loop) + @unittest.skipIf(ssl is None, 'No ssl module') def test_cancel_handshake(self): # Python issue #23197: cancelling an handshake must not raise an # exception or log an error, even if the handshake failed -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 08:33:40 2015 From: python-checkins at python.org (vinay.sajip) Date: Wed, 28 Jan 2015 07:33:40 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Added_a_loggin?= =?utf-8?q?g_cookbook_entry_on_customized_exception_formatting=2E?= Message-ID: <20150128073327.96084.52482@psf.io> https://hg.python.org/cpython/rev/1f07af946fe8 changeset: 94345:1f07af946fe8 branch: 2.7 parent: 94342:94d8524086bd user: Vinay Sajip date: Wed Jan 28 07:29:13 2015 +0000 summary: Added a logging cookbook entry on customized exception formatting. files: Doc/howto/logging-cookbook.rst | 55 ++++++++++++++++++++++ 1 files changed, 55 insertions(+), 0 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -1059,3 +1059,58 @@ information on how logging supports using user-defined objects in its configuration, and see the other cookbook recipe :ref:`custom-handlers` above. + +.. _custom-format-exception: + +Customized exception formatting +------------------------------- + +There might be times when you want to do customized exception formatting - for +argument's sake, let's say you want exactly one line per logged event, even +when exception information is present. You can do this with a custom formatter +class, as shown in the following example:: + + import logging + + class OneLineExceptionFormatter(logging.Formatter): + def formatException(self, exc_info): + """ + Format an exception so that it prints on a single line. + """ + result = super(OneLineExceptionFormatter, self).formatException(exc_info) + return repr(result) # or format into one line however you want to + + def format(self, record): + s = super(OneLineExceptionFormatter, self).format(record) + if record.exc_text: + s = s.replace('\n', '') + '|' + return s + + def configure_logging(): + fh = logging.FileHandler('output.txt', 'w') + f = OneLineExceptionFormatter('%(asctime)s|%(levelname)s|%(message)s|', + '%d/%m/%Y %H:%M:%S') + fh.setFormatter(f) + root = logging.getLogger() + root.setLevel(logging.DEBUG) + root.addHandler(fh) + + def main(): + configure_logging() + logging.info('Sample message') + try: + x = 1 / 0 + except ZeroDivisionError as e: + logging.exception('ZeroDivisionError: %s', e) + + if __name__ == '__main__': + main() + +When run, this produces a file with exactly two lines:: + + 28/01/2015 07:21:23|INFO|Sample message| + 28/01/2015 07:21:23|ERROR|ZeroDivisionError: integer division or modulo by zero|'Traceback (most recent call last):\n File "logtest7.py", line 30, in main\n x = 1 / 0\nZeroDivisionError: integer division or modulo by zero'| + +While the above treatment is simplistic, it points the way to how exception +information can be formatted to your liking. The :mod:`traceback` module may be +helpful for more specialized needs. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 08:33:40 2015 From: python-checkins at python.org (vinay.sajip) Date: Wed, 28 Jan 2015 07:33:40 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Added_a_loggin?= =?utf-8?q?g_cookbook_entry_on_customized_exception_formatting=2E?= Message-ID: <20150128073327.96076.87552@psf.io> https://hg.python.org/cpython/rev/eecaa42e0c06 changeset: 94346:eecaa42e0c06 branch: 3.4 parent: 94343:d61d1e73674f user: Vinay Sajip date: Wed Jan 28 07:32:38 2015 +0000 summary: Added a logging cookbook entry on customized exception formatting. files: Doc/howto/logging-cookbook.rst | 55 ++++++++++++++++++++++ 1 files changed, 55 insertions(+), 0 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -2033,3 +2033,58 @@ information on how logging supports using user-defined objects in its configuration, and see the other cookbook recipe :ref:`custom-handlers` above. + +.. _custom-format-exception: + +Customized exception formatting +------------------------------- + +There might be times when you want to do customized exception formatting - for +argument's sake, let's say you want exactly one line per logged event, even +when exception information is present. You can do this with a custom formatter +class, as shown in the following example:: + + import logging + + class OneLineExceptionFormatter(logging.Formatter): + def formatException(self, exc_info): + """ + Format an exception so that it prints on a single line. + """ + result = super(OneLineExceptionFormatter, self).formatException(exc_info) + return repr(result) # or format into one line however you want to + + def format(self, record): + s = super(OneLineExceptionFormatter, self).format(record) + if record.exc_text: + s = s.replace('\n', '') + '|' + return s + + def configure_logging(): + fh = logging.FileHandler('output.txt', 'w') + f = OneLineExceptionFormatter('%(asctime)s|%(levelname)s|%(message)s|', + '%d/%m/%Y %H:%M:%S') + fh.setFormatter(f) + root = logging.getLogger() + root.setLevel(logging.DEBUG) + root.addHandler(fh) + + def main(): + configure_logging() + logging.info('Sample message') + try: + x = 1 / 0 + except ZeroDivisionError as e: + logging.exception('ZeroDivisionError: %s', e) + + if __name__ == '__main__': + main() + +When run, this produces a file with exactly two lines:: + + 28/01/2015 07:21:23|INFO|Sample message| + 28/01/2015 07:21:23|ERROR|ZeroDivisionError: integer division or modulo by zero|'Traceback (most recent call last):\n File "logtest7.py", line 30, in main\n x = 1 / 0\nZeroDivisionError: integer division or modulo by zero'| + +While the above treatment is simplistic, it points the way to how exception +information can be formatted to your liking. The :mod:`traceback` module may be +helpful for more specialized needs. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 08:33:40 2015 From: python-checkins at python.org (vinay.sajip) Date: Wed, 28 Jan 2015 07:33:40 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merged_documentation_update_from_3=2E4=2E?= Message-ID: <20150128073327.106249.72441@psf.io> https://hg.python.org/cpython/rev/1df1fde8f360 changeset: 94347:1df1fde8f360 parent: 94344:3b920a778484 parent: 94346:eecaa42e0c06 user: Vinay Sajip date: Wed Jan 28 07:33:14 2015 +0000 summary: Merged documentation update from 3.4. files: Doc/howto/logging-cookbook.rst | 55 ++++++++++++++++++++++ 1 files changed, 55 insertions(+), 0 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -2033,3 +2033,58 @@ information on how logging supports using user-defined objects in its configuration, and see the other cookbook recipe :ref:`custom-handlers` above. + +.. _custom-format-exception: + +Customized exception formatting +------------------------------- + +There might be times when you want to do customized exception formatting - for +argument's sake, let's say you want exactly one line per logged event, even +when exception information is present. You can do this with a custom formatter +class, as shown in the following example:: + + import logging + + class OneLineExceptionFormatter(logging.Formatter): + def formatException(self, exc_info): + """ + Format an exception so that it prints on a single line. + """ + result = super(OneLineExceptionFormatter, self).formatException(exc_info) + return repr(result) # or format into one line however you want to + + def format(self, record): + s = super(OneLineExceptionFormatter, self).format(record) + if record.exc_text: + s = s.replace('\n', '') + '|' + return s + + def configure_logging(): + fh = logging.FileHandler('output.txt', 'w') + f = OneLineExceptionFormatter('%(asctime)s|%(levelname)s|%(message)s|', + '%d/%m/%Y %H:%M:%S') + fh.setFormatter(f) + root = logging.getLogger() + root.setLevel(logging.DEBUG) + root.addHandler(fh) + + def main(): + configure_logging() + logging.info('Sample message') + try: + x = 1 / 0 + except ZeroDivisionError as e: + logging.exception('ZeroDivisionError: %s', e) + + if __name__ == '__main__': + main() + +When run, this produces a file with exactly two lines:: + + 28/01/2015 07:21:23|INFO|Sample message| + 28/01/2015 07:21:23|ERROR|ZeroDivisionError: integer division or modulo by zero|'Traceback (most recent call last):\n File "logtest7.py", line 30, in main\n x = 1 / 0\nZeroDivisionError: integer division or modulo by zero'| + +While the above treatment is simplistic, it points the way to how exception +information can be formatted to your liking. The :mod:`traceback` module may be +helpful for more specialized needs. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 10:10:28 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 28 Jan 2015 09:10:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322079=3A_PyType=5FReady=28=29_now_checks_that_s?= =?utf-8?q?tatically_allocated_type_has?= Message-ID: <20150128091027.25853.94256@psf.io> https://hg.python.org/cpython/rev/eb26255e11f1 changeset: 94350:eb26255e11f1 parent: 94347:1df1fde8f360 parent: 94349:747855f29b9d user: Serhiy Storchaka date: Wed Jan 28 11:06:04 2015 +0200 summary: Issue #22079: PyType_Ready() now checks that statically allocated type has no dynamically allocated bases. files: Misc/NEWS | 3 +++ Objects/typeobject.c | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1519,6 +1519,9 @@ C API ----- +- Issue #22079: PyType_Ready() now checks that statically allocated type has + no dynamically allocated bases. + - Issue #22453: Removed non-documented macro PyObject_REPR(). - Issue #18395: Rename ``_Py_char2wchar()`` to :c:func:`Py_DecodeLocale`, diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -4683,6 +4683,20 @@ inherit_slots(type, (PyTypeObject *)b); } + /* All bases of statically allocated type should be statically allocated */ + if (!(type->tp_flags & Py_TPFLAGS_HEAPTYPE)) + for (i = 0; i < n; i++) { + PyObject *b = PyTuple_GET_ITEM(bases, i); + if (PyType_Check(b) && + (((PyTypeObject *)b)->tp_flags & Py_TPFLAGS_HEAPTYPE)) { + PyErr_Format(PyExc_TypeError, + "type '%.100s' is not dynamically allocated but " + "its base type '%.100s' is dynamically allocated", + type->tp_name, ((PyTypeObject *)b)->tp_name); + goto error; + } + } + /* Sanity check for tp_free. */ if (PyType_IS_GC(type) && (type->tp_flags & Py_TPFLAGS_BASETYPE) && (type->tp_free == NULL || type->tp_free == PyObject_Del)) { -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 10:10:28 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 28 Jan 2015 09:10:28 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIyMDc5?= =?utf-8?q?=3A_PyType=5FReady=28=29_now_checks_that_statically_allocated_t?= =?utf-8?q?ype_has?= Message-ID: <20150128091026.39292.49043@psf.io> https://hg.python.org/cpython/rev/c087ac6fc171 changeset: 94348:c087ac6fc171 branch: 2.7 parent: 94345:1f07af946fe8 user: Serhiy Storchaka date: Wed Jan 28 10:52:49 2015 +0200 summary: Issue #22079: PyType_Ready() now checks that statically allocated type has no dynamically allocated bases. files: Misc/NEWS | 6 ++++++ Objects/typeobject.c | 14 ++++++++++++++ 2 files changed, 20 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -117,6 +117,12 @@ - Issue #23212: Update 10.5 OS X installer build to use OpenSSL 1.0.1k. +C API +----- + +- Issue #22079: PyType_Ready() now checks that statically allocated type has + no dynamically allocated bases. + What's New in Python 2.7.9? =========================== diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -4073,6 +4073,20 @@ inherit_slots(type, (PyTypeObject *)b); } + /* All bases of statically allocated type should be statically allocated */ + if (!(type->tp_flags & Py_TPFLAGS_HEAPTYPE)) + for (i = 0; i < n; i++) { + PyObject *b = PyTuple_GET_ITEM(bases, i); + if (PyType_Check(b) && + (((PyTypeObject *)b)->tp_flags & Py_TPFLAGS_HEAPTYPE)) { + PyErr_Format(PyExc_TypeError, + "type '%.100s' is not dynamically allocated but " + "its base type '%.100s' is dynamically allocated", + type->tp_name, ((PyTypeObject *)b)->tp_name); + goto error; + } + } + /* Sanity check for tp_free. */ if (PyType_IS_GC(type) && (type->tp_flags & Py_TPFLAGS_BASETYPE) && (type->tp_free == NULL || type->tp_free == PyObject_Del)) { -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 10:10:28 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 28 Jan 2015 09:10:28 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyMDc5?= =?utf-8?q?=3A_PyType=5FReady=28=29_now_checks_that_statically_allocated_t?= =?utf-8?q?ype_has?= Message-ID: <20150128091026.96070.86811@psf.io> https://hg.python.org/cpython/rev/747855f29b9d changeset: 94349:747855f29b9d branch: 3.4 parent: 94346:eecaa42e0c06 user: Serhiy Storchaka date: Wed Jan 28 11:03:33 2015 +0200 summary: Issue #22079: PyType_Ready() now checks that statically allocated type has no dynamically allocated bases. files: Misc/NEWS | 7 +++++++ Objects/typeobject.c | 14 ++++++++++++++ 2 files changed, 21 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -332,6 +332,12 @@ - Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer. +C API +----- + +- Issue #22079: PyType_Ready() now checks that statically allocated type has + no dynamically allocated bases. + Documentation ------------- @@ -1136,6 +1142,7 @@ C API ----- + - Issue #20942: PyImport_ImportFrozenModuleObject() no longer sets __file__ to match what importlib does; this affects _frozen_importlib as well as any module loaded using imp.init_frozen(). diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -4680,6 +4680,20 @@ inherit_slots(type, (PyTypeObject *)b); } + /* All bases of statically allocated type should be statically allocated */ + if (!(type->tp_flags & Py_TPFLAGS_HEAPTYPE)) + for (i = 0; i < n; i++) { + PyObject *b = PyTuple_GET_ITEM(bases, i); + if (PyType_Check(b) && + (((PyTypeObject *)b)->tp_flags & Py_TPFLAGS_HEAPTYPE)) { + PyErr_Format(PyExc_TypeError, + "type '%.100s' is not dynamically allocated but " + "its base type '%.100s' is dynamically allocated", + type->tp_name, ((PyTypeObject *)b)->tp_name); + goto error; + } + } + /* Sanity check for tp_free. */ if (PyType_IS_GC(type) && (type->tp_flags & Py_TPFLAGS_BASETYPE) && (type->tp_free == NULL || type->tp_free == PyObject_Del)) { -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 16:12:21 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 28 Jan 2015 15:12:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fixed_memory_l?= =?utf-8?q?eak_in_marshal=2E?= Message-ID: <20150128151151.34404.34088@psf.io> https://hg.python.org/cpython/rev/8dfbaa2e8d26 changeset: 94351:8dfbaa2e8d26 branch: 3.4 parent: 94349:747855f29b9d user: Serhiy Storchaka date: Wed Jan 28 17:10:48 2015 +0200 summary: Fixed memory leak in marshal. files: Python/marshal.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Python/marshal.c b/Python/marshal.c --- a/Python/marshal.c +++ b/Python/marshal.c @@ -1522,8 +1522,10 @@ wf.depth = 0; wf.version = version; if (version >= 3) { - if ((wf.refs = PyDict_New()) == NULL) + if ((wf.refs = PyDict_New()) == NULL) { + Py_DECREF(wf.str); return NULL; + } } else wf.refs = NULL; w_object(x, &wf); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 16:12:21 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 28 Jan 2015 15:12:21 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Fixed_memory_leak_in_marshal=2E?= Message-ID: <20150128151151.34384.44874@psf.io> https://hg.python.org/cpython/rev/a0b2b5abf88c changeset: 94352:a0b2b5abf88c parent: 94350:eb26255e11f1 parent: 94351:8dfbaa2e8d26 user: Serhiy Storchaka date: Wed Jan 28 17:11:12 2015 +0200 summary: Fixed memory leak in marshal. files: Python/marshal.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Python/marshal.c b/Python/marshal.c --- a/Python/marshal.c +++ b/Python/marshal.c @@ -1520,8 +1520,10 @@ wf.depth = 0; wf.version = version; if (version >= 3) { - if ((wf.refs = PyDict_New()) == NULL) + if ((wf.refs = PyDict_New()) == NULL) { + Py_DECREF(wf.str); return NULL; + } } else wf.refs = NULL; w_object(x, &wf); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 18:07:09 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 28 Jan 2015 17:07:09 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_ifdef_our_way_?= =?utf-8?q?to_compatibility_with_old_openssl_=28closes_=2323335=29?= Message-ID: <20150128170654.39270.67856@psf.io> https://hg.python.org/cpython/rev/1addc4f0f10c changeset: 94354:1addc4f0f10c branch: 2.7 parent: 94348:c087ac6fc171 user: Benjamin Peterson date: Wed Jan 28 12:06:39 2015 -0500 summary: ifdef our way to compatibility with old openssl (closes #23335) files: Modules/_ssl.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2148,6 +2148,7 @@ Py_RETURN_NONE; } +#ifdef OPENSSL_NPN_NEGOTIATED static int do_protocol_selection(int alpn, unsigned char **out, unsigned char *outlen, const unsigned char *server_protocols, unsigned int server_protocols_len, @@ -2172,7 +2173,6 @@ return SSL_TLSEXT_ERR_OK; } -#ifdef OPENSSL_NPN_NEGOTIATED /* this callback gets passed to SSL_CTX_set_next_protos_advertise_cb */ static int _advertiseNPN_cb(SSL *s, -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 18:07:09 2015 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 28 Jan 2015 17:07:09 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_ifdef_our_way_to_compatibi?= =?utf-8?q?lity_with_old_openssl_=28closes_=2323335=29?= Message-ID: <20150128170653.106357.98009@psf.io> https://hg.python.org/cpython/rev/16f982f93a47 changeset: 94353:16f982f93a47 user: Benjamin Peterson date: Wed Jan 28 12:06:39 2015 -0500 summary: ifdef our way to compatibility with old openssl (closes #23335) files: Modules/_ssl.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2275,6 +2275,7 @@ Py_RETURN_NONE; } +#ifdef OPENSSL_NPN_NEGOTIATED static int do_protocol_selection(int alpn, unsigned char **out, unsigned char *outlen, const unsigned char *server_protocols, unsigned int server_protocols_len, @@ -2299,7 +2300,6 @@ return SSL_TLSEXT_ERR_OK; } -#ifdef OPENSSL_NPN_NEGOTIATED /* this callback gets passed to SSL_CTX_set_next_protos_advertise_cb */ static int _advertiseNPN_cb(SSL *s, -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 23:56:24 2015 From: python-checkins at python.org (donald.stufft) Date: Wed, 28 Jan 2015 22:56:24 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Update_setupto?= =?utf-8?q?ols_to_12=2E0=2E5_and_pip_to_6=2E0=2E7?= Message-ID: <20150128225621.106249.44054@psf.io> https://hg.python.org/cpython/rev/263344bb2107 changeset: 94355:263344bb2107 branch: 3.4 parent: 94351:8dfbaa2e8d26 user: Donald Stufft date: Wed Jan 28 17:56:15 2015 -0500 summary: Update setuptools to 12.0.5 and pip to 6.0.7 files: Lib/ensurepip/__init__.py | 4 ++-- Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl | Bin 5 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -8,9 +8,9 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "11.3.1" +_SETUPTOOLS_VERSION = "12.0.5" -_PIP_VERSION = "6.0.6" +_PIP_VERSION = "6.0.7" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl deleted file mode 100644 index e2be1055f375447e667e34ce2ce018ec41a12221..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..e01e8c4f44fbf99d9cb286f5a06b6689ff29d4d6 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl deleted file mode 100644 index 6892ff89b4f916c40a0044c18d4b41dbf73340e3..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..978cf6c43442faecdb4e8b84a3c2c0bc1bc0ec01 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 23:57:16 2015 From: python-checkins at python.org (donald.stufft) Date: Wed, 28 Jan 2015 22:57:16 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_into_default_-_Upgrade_setuptools_to_12=2E0?= =?utf-8?q?=2E5_and_pip_to_6=2E0=2E7?= Message-ID: <20150128225714.106346.92315@psf.io> https://hg.python.org/cpython/rev/e1eb09421b50 changeset: 94356:e1eb09421b50 parent: 94353:16f982f93a47 parent: 94355:263344bb2107 user: Donald Stufft date: Wed Jan 28 17:57:05 2015 -0500 summary: Merge 3.4 into default - Upgrade setuptools to 12.0.5 and pip to 6.0.7 files: Lib/ensurepip/__init__.py | 4 ++-- Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl | Bin 5 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -8,9 +8,9 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "11.3.1" +_SETUPTOOLS_VERSION = "12.0.5" -_PIP_VERSION = "6.0.6" +_PIP_VERSION = "6.0.7" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl deleted file mode 100644 index e2be1055f375447e667e34ce2ce018ec41a12221..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..e01e8c4f44fbf99d9cb286f5a06b6689ff29d4d6 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl deleted file mode 100644 index 6892ff89b4f916c40a0044c18d4b41dbf73340e3..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..978cf6c43442faecdb4e8b84a3c2c0bc1bc0ec01 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Jan 28 23:59:04 2015 From: python-checkins at python.org (donald.stufft) Date: Wed, 28 Jan 2015 22:59:04 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Upgrade_setupt?= =?utf-8?q?ools_to_12=2E0=2E5_and_pip_to_6=2E0=2E7?= Message-ID: <20150128225901.96078.97425@psf.io> https://hg.python.org/cpython/rev/a18444102268 changeset: 94357:a18444102268 branch: 2.7 parent: 94354:1addc4f0f10c user: Donald Stufft date: Wed Jan 28 17:58:53 2015 -0500 summary: Upgrade setuptools to 12.0.5 and pip to 6.0.7 files: Lib/ensurepip/__init__.py | 4 ++-- Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl | Bin 5 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -12,9 +12,9 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "11.3.1" +_SETUPTOOLS_VERSION = "12.0.5" -_PIP_VERSION = "6.0.6" +_PIP_VERSION = "6.0.7" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.6-py2.py3-none-any.whl deleted file mode 100644 index e2be1055f375447e667e34ce2ce018ec41a12221..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..e01e8c4f44fbf99d9cb286f5a06b6689ff29d4d6 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-11.3.1-py2.py3-none-any.whl deleted file mode 100644 index 6892ff89b4f916c40a0044c18d4b41dbf73340e3..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..978cf6c43442faecdb4e8b84a3c2c0bc1bc0ec01 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 00:47:05 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 28 Jan 2015 23:47:05 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogU1NM?= =?utf-8?q?_transports_now_clear_their_reference_to_the_waiter?= Message-ID: <20150128234705.96090.43091@psf.io> https://hg.python.org/cpython/rev/0ae40a8f10a1 changeset: 94359:0ae40a8f10a1 branch: 3.4 user: Victor Stinner date: Thu Jan 29 00:36:35 2015 +0100 summary: asyncio: SSL transports now clear their reference to the waiter * Rephrase also the comment explaining why the waiter is not awaken immediatly. * SSLProtocol.eof_received() doesn't instanciate ConnectionResetError exception directly, it will be done by Future.set_exception(). The exception is not used if the waiter was cancelled or if there is no waiter. files: Lib/asyncio/proactor_events.py | 2 +- Lib/asyncio/selector_events.py | 27 +++++++++++++-------- Lib/asyncio/sslproto.py | 20 ++++++++++----- Lib/asyncio/unix_events.py | 4 +- 4 files changed, 32 insertions(+), 21 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -38,7 +38,7 @@ self._server._attach() self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -581,7 +581,7 @@ self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def pause_reading(self): @@ -732,6 +732,16 @@ start_time = None self._on_handshake(start_time) + def _wakeup_waiter(self, exc=None): + if self._waiter is None: + return + if not self._waiter.cancelled(): + if exc is not None: + self._waiter.set_exception(exc) + else: + self._waiter.set_result(None) + self._waiter = None + def _on_handshake(self, start_time): try: self._sock.do_handshake() @@ -750,8 +760,7 @@ self._loop.remove_reader(self._sock_fd) self._loop.remove_writer(self._sock_fd) self._sock.close() - if self._waiter is not None and not self._waiter.cancelled(): - self._waiter.set_exception(exc) + self._wakeup_waiter(exc) if isinstance(exc, Exception): return else: @@ -774,9 +783,7 @@ "on matching the hostname", self, exc_info=True) self._sock.close() - if (self._waiter is not None - and not self._waiter.cancelled()): - self._waiter.set_exception(exc) + self._wakeup_waiter(exc) return # Add extra info that becomes available after handshake. @@ -789,10 +796,8 @@ self._write_wants_read = False self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) - if self._waiter is not None: - # wait until protocol.connection_made() has been called - self._loop.call_soon(self._waiter._set_result_unless_cancelled, - None) + # only wake up the waiter when connection_made() has been called + self._loop.call_soon(self._wakeup_waiter) if self._loop.get_debug(): dt = self._loop.time() - start_time @@ -924,7 +929,7 @@ self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def get_write_buffer_size(self): diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -418,6 +418,16 @@ self._in_shutdown = False self._transport = None + def _wakeup_waiter(self, exc=None): + if self._waiter is None: + return + if not self._waiter.cancelled(): + if exc is not None: + self._waiter.set_exception(exc) + else: + self._waiter.set_result(None) + self._waiter = None + def connection_made(self, transport): """Called when the low-level connection is made. @@ -490,8 +500,7 @@ if self._loop.get_debug(): logger.debug("%r received EOF", self) - if self._waiter is not None and not self._waiter.done(): - self._waiter.set_exception(ConnectionResetError()) + self._wakeup_waiter(ConnectionResetError) if not self._in_handshake: keep_open = self._app_protocol.eof_received() @@ -556,8 +565,7 @@ self, exc_info=True) self._transport.close() if isinstance(exc, Exception): - if self._waiter is not None and not self._waiter.cancelled(): - self._waiter.set_exception(exc) + self._wakeup_waiter(exc) return else: raise @@ -572,9 +580,7 @@ compression=sslobj.compression(), ) self._app_protocol.connection_made(self._app_transport) - if self._waiter is not None: - # wait until protocol.connection_made() has been called - self._waiter._set_result_unless_cancelled(None) + self._wakeup_waiter() self._session_established = True # In case transport.write() was already called. Don't call # immediatly _process_write_backlog(), but schedule it: diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -301,7 +301,7 @@ self._loop.add_reader(self._fileno, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): @@ -409,7 +409,7 @@ self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 00:47:05 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 28 Jan 2015 23:47:05 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?q?_SSLProtocol=2Eeof=5Freceived=28=29?= Message-ID: <20150128234705.34396.75256@psf.io> https://hg.python.org/cpython/rev/1da90ebae442 changeset: 94358:1da90ebae442 branch: 3.4 parent: 94355:263344bb2107 user: Victor Stinner date: Thu Jan 29 00:35:56 2015 +0100 summary: asyncio: Fix SSLProtocol.eof_received() Wake-up the waiter if it is not done yet. files: Lib/asyncio/sslproto.py | 4 + Lib/test/test_asyncio/test_sslproto.py | 40 ++++++++++--- 2 files changed, 33 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -489,6 +489,10 @@ try: if self._loop.get_debug(): logger.debug("%r received EOF", self) + + if self._waiter is not None and not self._waiter.done(): + self._waiter.set_exception(ConnectionResetError()) + if not self._in_handshake: keep_open = self._app_protocol.eof_received() if keep_open: diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -12,21 +12,36 @@ from asyncio import test_utils + at unittest.skipIf(ssl is None, 'No ssl module') class SslProtoHandshakeTests(test_utils.TestCase): def setUp(self): self.loop = asyncio.new_event_loop() self.set_event_loop(self.loop) - @unittest.skipIf(ssl is None, 'No ssl module') + def ssl_protocol(self, waiter=None): + sslcontext = test_utils.dummy_ssl_context() + app_proto = asyncio.Protocol() + return sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter) + + def connection_made(self, ssl_proto, do_handshake=None): + transport = mock.Mock() + sslpipe = mock.Mock() + sslpipe.shutdown.return_value = b'' + if do_handshake: + sslpipe.do_handshake.side_effect = do_handshake + else: + def mock_handshake(callback): + return [] + sslpipe.do_handshake.side_effect = mock_handshake + with mock.patch('asyncio.sslproto._SSLPipe', return_value=sslpipe): + ssl_proto.connection_made(transport) + def test_cancel_handshake(self): # Python issue #23197: cancelling an handshake must not raise an # exception or log an error, even if the handshake failed - sslcontext = test_utils.dummy_ssl_context() - app_proto = asyncio.Protocol() waiter = asyncio.Future(loop=self.loop) - ssl_proto = sslproto.SSLProtocol(self.loop, app_proto, sslcontext, - waiter) + ssl_proto = self.ssl_protocol(waiter) handshake_fut = asyncio.Future(loop=self.loop) def do_handshake(callback): @@ -36,12 +51,7 @@ return [] waiter.cancel() - transport = mock.Mock() - sslpipe = mock.Mock() - sslpipe.shutdown.return_value = b'' - sslpipe.do_handshake.side_effect = do_handshake - with mock.patch('asyncio.sslproto._SSLPipe', return_value=sslpipe): - ssl_proto.connection_made(transport) + self.connection_made(ssl_proto, do_handshake) with test_utils.disable_logger(): self.loop.run_until_complete(handshake_fut) @@ -49,6 +59,14 @@ # Close the transport ssl_proto._app_transport.close() + def test_eof_received_waiter(self): + waiter = asyncio.Future(loop=self.loop) + ssl_proto = self.ssl_protocol(waiter) + self.connection_made(ssl_proto) + ssl_proto.eof_received() + test_utils.run_briefly(self.loop) + self.assertIsInstance(waiter.exception(), ConnectionResetError) + if __name__ == '__main__': unittest.main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 00:47:05 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 28 Jan 2015 23:47:05 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?q?_=5FSelectorSocketTransport_constructor?= Message-ID: <20150128234705.25841.93936@psf.io> https://hg.python.org/cpython/rev/1b35bef31bf8 changeset: 94360:1b35bef31bf8 branch: 3.4 user: Victor Stinner date: Thu Jan 29 00:36:51 2015 +0100 summary: asyncio: Fix _SelectorSocketTransport constructor Only start reading when connection_made() has been called: protocol.data_received() must not be called before protocol.connection_made(). files: Lib/asyncio/selector_events.py | 4 +- Lib/test/test_asyncio/test_selector_events.py | 16 ++++++--- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -578,8 +578,10 @@ self._eof = False self._paused = False - self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._sock_fd, self._read_ready) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -59,6 +59,7 @@ def test_make_socket_transport(self): m = mock.Mock() self.loop.add_reader = mock.Mock() + self.loop.add_reader._is_coroutine = False transport = self.loop._make_socket_transport(m, asyncio.Protocol()) self.assertIsInstance(transport, _SelectorSocketTransport) close_transport(transport) @@ -67,6 +68,7 @@ def test_make_ssl_transport(self): m = mock.Mock() self.loop.add_reader = mock.Mock() + self.loop.add_reader._is_coroutine = False self.loop.add_writer = mock.Mock() self.loop.remove_reader = mock.Mock() self.loop.remove_writer = mock.Mock() @@ -770,20 +772,24 @@ return transport def test_ctor(self): - tr = self.socket_transport() + waiter = asyncio.Future(loop=self.loop) + tr = self.socket_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + self.loop.assert_reader(7, tr._read_ready) test_utils.run_briefly(self.loop) self.protocol.connection_made.assert_called_with(tr) def test_ctor_with_waiter(self): - fut = asyncio.Future(loop=self.loop) + waiter = asyncio.Future(loop=self.loop) + self.socket_transport(waiter=waiter) + self.loop.run_until_complete(waiter) - self.socket_transport(waiter=fut) - test_utils.run_briefly(self.loop) - self.assertIsNone(fut.result()) + self.assertIsNone(waiter.result()) def test_pause_resume_reading(self): tr = self.socket_transport() + test_utils.run_briefly(self.loop) self.assertFalse(tr._paused) self.loop.assert_reader(7, tr._read_ready) tr.pause_reading() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 00:56:38 2015 From: python-checkins at python.org (victor.stinner) Date: Wed, 28 Jan 2015 23:56:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150128235637.106271.13038@psf.io> https://hg.python.org/cpython/rev/08c67d4f8e36 changeset: 94361:08c67d4f8e36 parent: 94356:e1eb09421b50 parent: 94360:1b35bef31bf8 user: Victor Stinner date: Thu Jan 29 00:55:46 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/proactor_events.py | 2 +- Lib/asyncio/selector_events.py | 31 ++++--- Lib/asyncio/sslproto.py | 20 +++- Lib/asyncio/unix_events.py | 4 +- Lib/test/test_asyncio/test_selector_events.py | 16 ++- Lib/test/test_asyncio/test_sslproto.py | 40 +++++++-- 6 files changed, 77 insertions(+), 36 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -38,7 +38,7 @@ self._server._attach() self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -578,10 +578,12 @@ self._eof = False self._paused = False - self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._sock_fd, self._read_ready) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def pause_reading(self): @@ -732,6 +734,16 @@ start_time = None self._on_handshake(start_time) + def _wakeup_waiter(self, exc=None): + if self._waiter is None: + return + if not self._waiter.cancelled(): + if exc is not None: + self._waiter.set_exception(exc) + else: + self._waiter.set_result(None) + self._waiter = None + def _on_handshake(self, start_time): try: self._sock.do_handshake() @@ -750,8 +762,7 @@ self._loop.remove_reader(self._sock_fd) self._loop.remove_writer(self._sock_fd) self._sock.close() - if self._waiter is not None and not self._waiter.cancelled(): - self._waiter.set_exception(exc) + self._wakeup_waiter(exc) if isinstance(exc, Exception): return else: @@ -774,9 +785,7 @@ "on matching the hostname", self, exc_info=True) self._sock.close() - if (self._waiter is not None - and not self._waiter.cancelled()): - self._waiter.set_exception(exc) + self._wakeup_waiter(exc) return # Add extra info that becomes available after handshake. @@ -789,10 +798,8 @@ self._write_wants_read = False self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) - if self._waiter is not None: - # wait until protocol.connection_made() has been called - self._loop.call_soon(self._waiter._set_result_unless_cancelled, - None) + # only wake up the waiter when connection_made() has been called + self._loop.call_soon(self._wakeup_waiter) if self._loop.get_debug(): dt = self._loop.time() - start_time @@ -924,7 +931,7 @@ self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def get_write_buffer_size(self): diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -418,6 +418,16 @@ self._in_shutdown = False self._transport = None + def _wakeup_waiter(self, exc=None): + if self._waiter is None: + return + if not self._waiter.cancelled(): + if exc is not None: + self._waiter.set_exception(exc) + else: + self._waiter.set_result(None) + self._waiter = None + def connection_made(self, transport): """Called when the low-level connection is made. @@ -489,6 +499,9 @@ try: if self._loop.get_debug(): logger.debug("%r received EOF", self) + + self._wakeup_waiter(ConnectionResetError) + if not self._in_handshake: keep_open = self._app_protocol.eof_received() if keep_open: @@ -552,8 +565,7 @@ self, exc_info=True) self._transport.close() if isinstance(exc, Exception): - if self._waiter is not None and not self._waiter.cancelled(): - self._waiter.set_exception(exc) + self._wakeup_waiter(exc) return else: raise @@ -568,9 +580,7 @@ compression=sslobj.compression(), ) self._app_protocol.connection_made(self._app_transport) - if self._waiter is not None: - # wait until protocol.connection_made() has been called - self._waiter._set_result_unless_cancelled(None) + self._wakeup_waiter() self._session_established = True # In case transport.write() was already called. Don't call # immediatly _process_write_backlog(), but schedule it: diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -301,7 +301,7 @@ self._loop.add_reader(self._fileno, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): @@ -409,7 +409,7 @@ self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: - # wait until protocol.connection_made() has been called + # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) def __repr__(self): diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -59,6 +59,7 @@ def test_make_socket_transport(self): m = mock.Mock() self.loop.add_reader = mock.Mock() + self.loop.add_reader._is_coroutine = False transport = self.loop._make_socket_transport(m, asyncio.Protocol()) self.assertIsInstance(transport, _SelectorSocketTransport) close_transport(transport) @@ -67,6 +68,7 @@ def test_make_ssl_transport(self): m = mock.Mock() self.loop.add_reader = mock.Mock() + self.loop.add_reader._is_coroutine = False self.loop.add_writer = mock.Mock() self.loop.remove_reader = mock.Mock() self.loop.remove_writer = mock.Mock() @@ -770,20 +772,24 @@ return transport def test_ctor(self): - tr = self.socket_transport() + waiter = asyncio.Future(loop=self.loop) + tr = self.socket_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + self.loop.assert_reader(7, tr._read_ready) test_utils.run_briefly(self.loop) self.protocol.connection_made.assert_called_with(tr) def test_ctor_with_waiter(self): - fut = asyncio.Future(loop=self.loop) + waiter = asyncio.Future(loop=self.loop) + self.socket_transport(waiter=waiter) + self.loop.run_until_complete(waiter) - self.socket_transport(waiter=fut) - test_utils.run_briefly(self.loop) - self.assertIsNone(fut.result()) + self.assertIsNone(waiter.result()) def test_pause_resume_reading(self): tr = self.socket_transport() + test_utils.run_briefly(self.loop) self.assertFalse(tr._paused) self.loop.assert_reader(7, tr._read_ready) tr.pause_reading() diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -12,21 +12,36 @@ from asyncio import test_utils + at unittest.skipIf(ssl is None, 'No ssl module') class SslProtoHandshakeTests(test_utils.TestCase): def setUp(self): self.loop = asyncio.new_event_loop() self.set_event_loop(self.loop) - @unittest.skipIf(ssl is None, 'No ssl module') + def ssl_protocol(self, waiter=None): + sslcontext = test_utils.dummy_ssl_context() + app_proto = asyncio.Protocol() + return sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter) + + def connection_made(self, ssl_proto, do_handshake=None): + transport = mock.Mock() + sslpipe = mock.Mock() + sslpipe.shutdown.return_value = b'' + if do_handshake: + sslpipe.do_handshake.side_effect = do_handshake + else: + def mock_handshake(callback): + return [] + sslpipe.do_handshake.side_effect = mock_handshake + with mock.patch('asyncio.sslproto._SSLPipe', return_value=sslpipe): + ssl_proto.connection_made(transport) + def test_cancel_handshake(self): # Python issue #23197: cancelling an handshake must not raise an # exception or log an error, even if the handshake failed - sslcontext = test_utils.dummy_ssl_context() - app_proto = asyncio.Protocol() waiter = asyncio.Future(loop=self.loop) - ssl_proto = sslproto.SSLProtocol(self.loop, app_proto, sslcontext, - waiter) + ssl_proto = self.ssl_protocol(waiter) handshake_fut = asyncio.Future(loop=self.loop) def do_handshake(callback): @@ -36,12 +51,7 @@ return [] waiter.cancel() - transport = mock.Mock() - sslpipe = mock.Mock() - sslpipe.shutdown.return_value = b'' - sslpipe.do_handshake.side_effect = do_handshake - with mock.patch('asyncio.sslproto._SSLPipe', return_value=sslpipe): - ssl_proto.connection_made(transport) + self.connection_made(ssl_proto, do_handshake) with test_utils.disable_logger(): self.loop.run_until_complete(handshake_fut) @@ -49,6 +59,14 @@ # Close the transport ssl_proto._app_transport.close() + def test_eof_received_waiter(self): + waiter = asyncio.Future(loop=self.loop) + ssl_proto = self.ssl_protocol(waiter) + self.connection_made(ssl_proto) + ssl_proto.eof_received() + test_utils.run_briefly(self.loop) + self.assertIsInstance(waiter.exception(), ConnectionResetError) + if __name__ == '__main__': unittest.main() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 01:08:27 2015 From: python-checkins at python.org (gregory.p.smith) Date: Thu, 29 Jan 2015 00:08:27 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQWx3YXlzICNkZWZp?= =?utf-8?q?ne_=5FPyLong=5FFromDev_as_we_always_need_it_to_compile_rather_t?= =?utf-8?q?han?= Message-ID: <20150129000823.106309.79058@psf.io> https://hg.python.org/cpython/rev/8fb1abedd3d5 changeset: 94362:8fb1abedd3d5 branch: 3.4 parent: 94360:1b35bef31bf8 user: Gregory P. Smith date: Wed Jan 28 16:07:52 2015 -0800 summary: Always #define _PyLong_FromDev as we always need it to compile rather than only defining it when HAVE_MKNOD && HAVE_MAKEDEV are true. This "oops" issue reported by John E. Malmberg on core-mentorship. (what kinds of systems don't HAVE_MKNOD && HAVE_MAKEDEV?) files: Modules/posixmodule.c | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -623,6 +623,13 @@ #endif /* MS_WINDOWS */ +#ifdef HAVE_LONG_LONG +# define _PyLong_FromDev PyLong_FromLongLong +#else +# define _PyLong_FromDev PyLong_FromLong +#endif + + #if defined(HAVE_MKNOD) && defined(HAVE_MAKEDEV) static int _Py_Dev_Converter(PyObject *obj, void *p) @@ -636,14 +643,7 @@ return 0; return 1; } - -#ifdef HAVE_LONG_LONG -# define _PyLong_FromDev PyLong_FromLongLong -#else -# define _PyLong_FromDev PyLong_FromLong -#endif - -#endif +#endif /* HAVE_MKNOD && HAVE_MAKEDEV */ #ifdef AT_FDCWD -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 01:08:27 2015 From: python-checkins at python.org (gregory.p.smith) Date: Thu, 29 Jan 2015 00:08:27 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Always_=23define_=5FPyLong=5FFromDev_as_we_always_need_i?= =?utf-8?q?t_to_compile_rather_than?= Message-ID: <20150129000823.25855.93781@psf.io> https://hg.python.org/cpython/rev/630e128b85ed changeset: 94363:630e128b85ed parent: 94361:08c67d4f8e36 parent: 94362:8fb1abedd3d5 user: Gregory P. Smith date: Wed Jan 28 16:08:07 2015 -0800 summary: Always #define _PyLong_FromDev as we always need it to compile rather than only defining it when HAVE_MKNOD && HAVE_MAKEDEV are true. This "oops" issue reported by John E. Malmberg on core-mentorship. (what kinds of systems don't HAVE_MKNOD && HAVE_MAKEDEV?) files: Modules/posixmodule.c | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -617,6 +617,13 @@ #endif /* MS_WINDOWS */ +#ifdef HAVE_LONG_LONG +# define _PyLong_FromDev PyLong_FromLongLong +#else +# define _PyLong_FromDev PyLong_FromLong +#endif + + #if defined(HAVE_MKNOD) && defined(HAVE_MAKEDEV) static int _Py_Dev_Converter(PyObject *obj, void *p) @@ -630,14 +637,7 @@ return 0; return 1; } - -#ifdef HAVE_LONG_LONG -# define _PyLong_FromDev PyLong_FromLongLong -#else -# define _PyLong_FromDev PyLong_FromLong -#endif - -#endif +#endif /* HAVE_MKNOD && HAVE_MAKEDEV */ #ifdef AT_FDCWD -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 02:16:42 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 01:16:42 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150129011641.96098.59683@psf.io> https://hg.python.org/cpython/rev/fee78636a3b1 changeset: 94365:fee78636a3b1 parent: 94363:630e128b85ed parent: 94364:44f90017eee3 user: Victor Stinner date: Thu Jan 29 02:15:35 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_subprocess.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -117,12 +117,15 @@ proc.stderr.close() if proc.stdin: proc.stdin.close() + try: proc.kill() except ProcessLookupError: pass self._returncode = proc.wait() + self.close() + @coroutine def _post_init(self): try: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 02:16:42 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 01:16:42 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQmFz?= =?utf-8?q?eSubprocessTransport=2E=5Fkill=5Fwait=28=29_now_also_call_close?= =?utf-8?b?KCk=?= Message-ID: <20150129011640.96096.31979@psf.io> https://hg.python.org/cpython/rev/44f90017eee3 changeset: 94364:44f90017eee3 branch: 3.4 parent: 94362:8fb1abedd3d5 user: Victor Stinner date: Thu Jan 29 02:14:30 2015 +0100 summary: asyncio: BaseSubprocessTransport._kill_wait() now also call close() close() closes pipes, which is not None yet by _kill_wait(). files: Lib/asyncio/base_subprocess.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -117,12 +117,15 @@ proc.stderr.close() if proc.stdin: proc.stdin.close() + try: proc.kill() except ProcessLookupError: pass self._returncode = proc.wait() + self.close() + @coroutine def _post_init(self): try: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 02:58:13 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 01:58:13 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogc3lu?= =?utf-8?q?c_with_Tulip?= Message-ID: <20150129015809.34386.97022@psf.io> https://hg.python.org/cpython/rev/65ee4dec4f1d changeset: 94366:65ee4dec4f1d branch: 3.4 parent: 94364:44f90017eee3 user: Victor Stinner date: Thu Jan 29 02:56:05 2015 +0100 summary: asyncio: sync with Tulip * _SelectorTransport constructor: extra parameter is now optional * Fix _SelectorDatagramTransport constructor. Only start reading after connection_made() has been called. * Fix _SelectorSslTransport.close(). Don't call protocol.connection_lost() if protocol.connection_made() was not called yet: if the SSL handshake failed or is still in progress. The close() method can be called if the creation of the connection is cancelled, by a timeout for example. files: Lib/asyncio/selector_events.py | 13 ++++++-- Lib/test/test_asyncio/test_selector_events.py | 15 +++++++++- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -467,7 +467,7 @@ _buffer_factory = bytearray # Constructs initial value for self._buffer. - def __init__(self, loop, sock, protocol, extra, server=None): + def __init__(self, loop, sock, protocol, extra=None, server=None): super().__init__(extra, loop) self._extra['socket'] = sock self._extra['sockname'] = sock.getsockname() @@ -479,6 +479,7 @@ self._sock = sock self._sock_fd = sock.fileno() self._protocol = protocol + self._protocol_connected = True self._server = server self._buffer = self._buffer_factory() self._conn_lost = 0 # Set when call to connection_lost scheduled. @@ -555,7 +556,8 @@ def _call_connection_lost(self, exc): try: - self._protocol.connection_lost(exc) + if self._protocol_connected: + self._protocol.connection_lost(exc) finally: self._sock.close() self._sock = None @@ -718,6 +720,8 @@ sslsock = sslcontext.wrap_socket(rawsock, **wrap_kwargs) super().__init__(loop, sslsock, protocol, extra, server) + # the protocol connection is only made after the SSL handshake + self._protocol_connected = False self._server_hostname = server_hostname self._waiter = waiter @@ -797,6 +801,7 @@ self._read_wants_write = False self._write_wants_read = False self._loop.add_reader(self._sock_fd, self._read_ready) + self._protocol_connected = True self._loop.call_soon(self._protocol.connection_made, self) # only wake up the waiter when connection_made() has been called self._loop.call_soon(self._wakeup_waiter) @@ -928,8 +933,10 @@ waiter=None, extra=None): super().__init__(loop, sock, protocol, extra) self._address = address - self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._sock_fd, self._read_ready) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1427,7 +1427,7 @@ self.assertFalse(tr.can_write_eof()) self.assertRaises(NotImplementedError, tr.write_eof) - def test_close(self): + def check_close(self): tr = self._make_one() tr.close() @@ -1439,6 +1439,19 @@ self.assertEqual(tr._conn_lost, 1) self.assertEqual(1, self.loop.remove_reader_count[1]) + test_utils.run_briefly(self.loop) + + def test_close(self): + self.check_close() + self.assertTrue(self.protocol.connection_made.called) + self.assertTrue(self.protocol.connection_lost.called) + + def test_close_not_connected(self): + self.sslsock.do_handshake.side_effect = ssl.SSLWantReadError + self.check_close() + self.assertFalse(self.protocol.connection_made.called) + self.assertFalse(self.protocol.connection_lost.called) + @unittest.skipIf(ssl is None, 'No SSL support') def test_server_hostname(self): self.ssl_transport(server_hostname='localhost') -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 02:58:13 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 01:58:13 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150129015810.25849.97534@psf.io> https://hg.python.org/cpython/rev/25028b0e1183 changeset: 94367:25028b0e1183 parent: 94365:fee78636a3b1 parent: 94366:65ee4dec4f1d user: Victor Stinner date: Thu Jan 29 02:57:10 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/selector_events.py | 13 ++++++-- Lib/test/test_asyncio/test_selector_events.py | 15 +++++++++- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -467,7 +467,7 @@ _buffer_factory = bytearray # Constructs initial value for self._buffer. - def __init__(self, loop, sock, protocol, extra, server=None): + def __init__(self, loop, sock, protocol, extra=None, server=None): super().__init__(extra, loop) self._extra['socket'] = sock self._extra['sockname'] = sock.getsockname() @@ -479,6 +479,7 @@ self._sock = sock self._sock_fd = sock.fileno() self._protocol = protocol + self._protocol_connected = True self._server = server self._buffer = self._buffer_factory() self._conn_lost = 0 # Set when call to connection_lost scheduled. @@ -555,7 +556,8 @@ def _call_connection_lost(self, exc): try: - self._protocol.connection_lost(exc) + if self._protocol_connected: + self._protocol.connection_lost(exc) finally: self._sock.close() self._sock = None @@ -718,6 +720,8 @@ sslsock = sslcontext.wrap_socket(rawsock, **wrap_kwargs) super().__init__(loop, sslsock, protocol, extra, server) + # the protocol connection is only made after the SSL handshake + self._protocol_connected = False self._server_hostname = server_hostname self._waiter = waiter @@ -797,6 +801,7 @@ self._read_wants_write = False self._write_wants_read = False self._loop.add_reader(self._sock_fd, self._read_ready) + self._protocol_connected = True self._loop.call_soon(self._protocol.connection_made, self) # only wake up the waiter when connection_made() has been called self._loop.call_soon(self._wakeup_waiter) @@ -928,8 +933,10 @@ waiter=None, extra=None): super().__init__(loop, sock, protocol, extra) self._address = address - self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._sock_fd, self._read_ready) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1427,7 +1427,7 @@ self.assertFalse(tr.can_write_eof()) self.assertRaises(NotImplementedError, tr.write_eof) - def test_close(self): + def check_close(self): tr = self._make_one() tr.close() @@ -1439,6 +1439,19 @@ self.assertEqual(tr._conn_lost, 1) self.assertEqual(1, self.loop.remove_reader_count[1]) + test_utils.run_briefly(self.loop) + + def test_close(self): + self.check_close() + self.assertTrue(self.protocol.connection_made.called) + self.assertTrue(self.protocol.connection_lost.called) + + def test_close_not_connected(self): + self.sslsock.do_handshake.side_effect = ssl.SSLWantReadError + self.check_close() + self.assertFalse(self.protocol.connection_made.called) + self.assertFalse(self.protocol.connection_lost.called) + @unittest.skipIf(ssl is None, 'No SSL support') def test_server_hostname(self): self.ssl_transport(server_hostname='localhost') -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Thu Jan 29 09:06:27 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 29 Jan 2015 09:06:27 +0100 Subject: [Python-checkins] Daily reference leaks (25028b0e1183): sum=9 Message-ID: results for 25028b0e1183 on branch "default" -------------------------------------------- test_asyncio leaked [3, 0, 3] memory blocks, sum=6 test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogCIJoGx', '-x'] From python-checkins at python.org Thu Jan 29 13:34:56 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 12:34:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbyBkb2M6?= =?utf-8?q?_document_Protocol_state_machine?= Message-ID: <20150129123334.34390.45700@psf.io> https://hg.python.org/cpython/rev/4fe1303bf531 changeset: 94368:4fe1303bf531 branch: 3.4 parent: 94366:65ee4dec4f1d user: Victor Stinner date: Thu Jan 29 13:33:15 2015 +0100 summary: asyncio doc: document Protocol state machine files: Doc/library/asyncio-protocol.rst | 8 ++++++++ Lib/asyncio/protocols.py | 5 +++++ 2 files changed, 13 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -374,6 +374,14 @@ a connection. However, :meth:`eof_received` is called at most once and, if called, :meth:`data_received` won't be called after it. +State machine: + + start -> :meth:`~BaseProtocol.connection_made` + [-> :meth:`~Protocol.data_received` \*] + [-> :meth:`~Protocol.eof_received` ?] + -> :meth:`~BaseProtocol.connection_lost` -> end + + Datagram protocols ------------------ diff --git a/Lib/asyncio/protocols.py b/Lib/asyncio/protocols.py --- a/Lib/asyncio/protocols.py +++ b/Lib/asyncio/protocols.py @@ -78,6 +78,11 @@ State machine of calls: start -> CM [-> DR*] [-> ER?] -> CL -> end + + * CM: connection_made() + * DR: data_received() + * ER: eof_received() + * CL: connection_lost() """ def data_received(self, data): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 13:34:56 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 12:34:56 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?= Message-ID: <20150129123335.25861.78391@psf.io> https://hg.python.org/cpython/rev/647e6176d7d7 changeset: 94369:647e6176d7d7 parent: 94367:25028b0e1183 parent: 94368:4fe1303bf531 user: Victor Stinner date: Thu Jan 29 13:33:28 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-protocol.rst | 8 ++++++++ Lib/asyncio/protocols.py | 5 +++++ 2 files changed, 13 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -374,6 +374,14 @@ a connection. However, :meth:`eof_received` is called at most once and, if called, :meth:`data_received` won't be called after it. +State machine: + + start -> :meth:`~BaseProtocol.connection_made` + [-> :meth:`~Protocol.data_received` \*] + [-> :meth:`~Protocol.eof_received` ?] + -> :meth:`~BaseProtocol.connection_lost` -> end + + Datagram protocols ------------------ diff --git a/Lib/asyncio/protocols.py b/Lib/asyncio/protocols.py --- a/Lib/asyncio/protocols.py +++ b/Lib/asyncio/protocols.py @@ -78,6 +78,11 @@ State machine of calls: start -> CM [-> DR*] [-> ER?] -> CL -> end + + * CM: connection_made() + * DR: data_received() + * ER: eof_received() + * CL: connection_lost() """ def data_received(self, data): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 14:18:36 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 13:18:36 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogc3lu?= =?utf-8?q?c_with_Tulip?= Message-ID: <20150129131705.96090.1945@psf.io> https://hg.python.org/cpython/rev/c4fd6df9aea6 changeset: 94370:c4fd6df9aea6 branch: 3.4 parent: 94368:4fe1303bf531 user: Victor Stinner date: Thu Jan 29 14:15:19 2015 +0100 summary: asyncio: sync with Tulip * Cleanup gather(): use cancelled() method instead of using private Future attribute * Fix _UnixReadPipeTransport and _UnixWritePipeTransport. Only start reading when connection_made() has been called. * Issue #23333: Fix BaseSelectorEventLoop._accept_connection(). Close the transport on error. In debug mode, log errors using call_exception_handler() files: Lib/asyncio/selector_events.py | 44 +++++++++- Lib/asyncio/tasks.py | 2 +- Lib/asyncio/unix_events.py | 17 ++- Lib/test/test_asyncio/test_events.py | 37 ++++++--- Lib/test/test_asyncio/test_unix_events.py | 31 +++---- 5 files changed, 86 insertions(+), 45 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -22,6 +22,7 @@ from . import selectors from . import transports from . import sslproto +from .coroutines import coroutine from .log import logger @@ -181,16 +182,47 @@ else: raise # The event loop will catch, log and ignore it. else: + extra = {'peername': addr} + accept = self._accept_connection2(protocol_factory, conn, extra, + sslcontext, server) + self.create_task(accept) + + @coroutine + def _accept_connection2(self, protocol_factory, conn, extra, + sslcontext=None, server=None): + protocol = None + transport = None + try: protocol = protocol_factory() + waiter = futures.Future(loop=self) if sslcontext: - self._make_ssl_transport( - conn, protocol, sslcontext, - server_side=True, extra={'peername': addr}, server=server) + transport = self._make_ssl_transport( + conn, protocol, sslcontext, waiter=waiter, + server_side=True, extra=extra, server=server) else: - self._make_socket_transport( - conn, protocol , extra={'peername': addr}, + transport = self._make_socket_transport( + conn, protocol, waiter=waiter, extra=extra, server=server) - # It's now up to the protocol to handle the connection. + + try: + yield from waiter + except: + transport.close() + raise + + # It's now up to the protocol to handle the connection. + except Exception as exc: + if self.get_debug(): + context = { + 'message': ('Error on transport creation ' + 'for incoming connection'), + 'exception': exc, + } + if protocol is not None: + context['protocol'] = protocol + if transport is not None: + context['transport'] = transport + self.call_exception_handler(context) def add_reader(self, fd, callback, *args): """Add a reader callback.""" diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -592,7 +592,7 @@ fut.exception() return - if fut._state == futures._CANCELLED: + if fut.cancelled(): res = futures.CancelledError() if not return_exceptions: outer.set_exception(res) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -298,8 +298,10 @@ _set_nonblocking(self._fileno) self._protocol = protocol self._closing = False - self._loop.add_reader(self._fileno, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._fileno, self._read_ready) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) @@ -401,13 +403,16 @@ self._conn_lost = 0 self._closing = False # Set when close() or write_eof() called. - # On AIX, the reader trick only works for sockets. - # On other platforms it works for pipes and sockets. - # (Exception: OS X 10.4? Issue #19294.) + self._loop.call_soon(self._protocol.connection_made, self) + + # On AIX, the reader trick (to be notified when the read end of the + # socket is closed) only works for sockets. On other platforms it + # works for pipes and sockets. (Exception: OS X 10.4? Issue #19294.) if is_socket or not sys.platform.startswith("aix"): - self._loop.add_reader(self._fileno, self._read_ready) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._fileno, self._read_ready) - self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -886,13 +886,18 @@ if hasattr(sslcontext_client, 'check_hostname'): sslcontext_client.check_hostname = True + # no CA loaded f_c = self.loop.create_connection(MyProto, host, port, ssl=sslcontext_client) - with test_utils.disable_logger(): - with self.assertRaisesRegex(ssl.SSLError, - 'certificate verify failed '): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.SSLError, + 'certificate verify failed '): + self.loop.run_until_complete(f_c) + + # execute the loop to log the connection error + test_utils.run_briefly(self.loop) # close connection self.assertIsNone(proto.transport) @@ -919,15 +924,20 @@ f_c = self.loop.create_unix_connection(MyProto, path, ssl=sslcontext_client, server_hostname='invalid') - with test_utils.disable_logger(): - with self.assertRaisesRegex(ssl.SSLError, - 'certificate verify failed '): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.SSLError, + 'certificate verify failed '): + self.loop.run_until_complete(f_c) + + # execute the loop to log the connection error + test_utils.run_briefly(self.loop) # close connection self.assertIsNone(proto.transport) server.close() + def test_legacy_create_unix_server_ssl_verify_failed(self): with test_utils.force_legacy_ssl_support(): self.test_create_unix_server_ssl_verify_failed() @@ -949,11 +959,12 @@ # incorrect server_hostname f_c = self.loop.create_connection(MyProto, host, port, ssl=sslcontext_client) - with test_utils.disable_logger(): - with self.assertRaisesRegex( - ssl.CertificateError, - "hostname '127.0.0.1' doesn't match 'localhost'"): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex( + ssl.CertificateError, + "hostname '127.0.0.1' doesn't match 'localhost'"): + self.loop.run_until_complete(f_c) # close connection proto.transport.close() diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -350,16 +350,13 @@ return transport def test_ctor(self): - tr = self.read_pipe_transport() + waiter = asyncio.Future(loop=self.loop) + tr = self.read_pipe_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.protocol.connection_made.assert_called_with(tr) self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.protocol.connection_made.assert_called_with(tr) - - def test_ctor_with_waiter(self): - fut = asyncio.Future(loop=self.loop) - tr = self.read_pipe_transport(waiter=fut) - test_utils.run_briefly(self.loop) - self.assertIsNone(fut.result()) + self.assertIsNone(waiter.result()) @mock.patch('os.read') def test__read_ready(self, m_read): @@ -502,17 +499,13 @@ return transport def test_ctor(self): - tr = self.write_pipe_transport() + waiter = asyncio.Future(loop=self.loop) + tr = self.write_pipe_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.protocol.connection_made.assert_called_with(tr) self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.protocol.connection_made.assert_called_with(tr) - - def test_ctor_with_waiter(self): - fut = asyncio.Future(loop=self.loop) - tr = self.write_pipe_transport(waiter=fut) - self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.assertEqual(None, fut.result()) + self.assertEqual(None, waiter.result()) def test_can_write_eof(self): tr = self.write_pipe_transport() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 14:18:36 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 13:18:36 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150129131705.96080.55780@psf.io> https://hg.python.org/cpython/rev/497c9e57a54e changeset: 94371:497c9e57a54e parent: 94369:647e6176d7d7 parent: 94370:c4fd6df9aea6 user: Victor Stinner date: Thu Jan 29 14:15:42 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/selector_events.py | 44 +++++++++- Lib/asyncio/tasks.py | 2 +- Lib/asyncio/unix_events.py | 17 ++- Lib/test/test_asyncio/test_events.py | 37 ++++++--- Lib/test/test_asyncio/test_unix_events.py | 31 +++---- 5 files changed, 86 insertions(+), 45 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -22,6 +22,7 @@ from . import selectors from . import transports from . import sslproto +from .coroutines import coroutine from .log import logger @@ -181,16 +182,47 @@ else: raise # The event loop will catch, log and ignore it. else: + extra = {'peername': addr} + accept = self._accept_connection2(protocol_factory, conn, extra, + sslcontext, server) + self.create_task(accept) + + @coroutine + def _accept_connection2(self, protocol_factory, conn, extra, + sslcontext=None, server=None): + protocol = None + transport = None + try: protocol = protocol_factory() + waiter = futures.Future(loop=self) if sslcontext: - self._make_ssl_transport( - conn, protocol, sslcontext, - server_side=True, extra={'peername': addr}, server=server) + transport = self._make_ssl_transport( + conn, protocol, sslcontext, waiter=waiter, + server_side=True, extra=extra, server=server) else: - self._make_socket_transport( - conn, protocol , extra={'peername': addr}, + transport = self._make_socket_transport( + conn, protocol, waiter=waiter, extra=extra, server=server) - # It's now up to the protocol to handle the connection. + + try: + yield from waiter + except: + transport.close() + raise + + # It's now up to the protocol to handle the connection. + except Exception as exc: + if self.get_debug(): + context = { + 'message': ('Error on transport creation ' + 'for incoming connection'), + 'exception': exc, + } + if protocol is not None: + context['protocol'] = protocol + if transport is not None: + context['transport'] = transport + self.call_exception_handler(context) def add_reader(self, fd, callback, *args): """Add a reader callback.""" diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -592,7 +592,7 @@ fut.exception() return - if fut._state == futures._CANCELLED: + if fut.cancelled(): res = futures.CancelledError() if not return_exceptions: outer.set_exception(res) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -298,8 +298,10 @@ _set_nonblocking(self._fileno) self._protocol = protocol self._closing = False - self._loop.add_reader(self._fileno, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._fileno, self._read_ready) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) @@ -401,13 +403,16 @@ self._conn_lost = 0 self._closing = False # Set when close() or write_eof() called. - # On AIX, the reader trick only works for sockets. - # On other platforms it works for pipes and sockets. - # (Exception: OS X 10.4? Issue #19294.) + self._loop.call_soon(self._protocol.connection_made, self) + + # On AIX, the reader trick (to be notified when the read end of the + # socket is closed) only works for sockets. On other platforms it + # works for pipes and sockets. (Exception: OS X 10.4? Issue #19294.) if is_socket or not sys.platform.startswith("aix"): - self._loop.add_reader(self._fileno, self._read_ready) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._fileno, self._read_ready) - self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -886,13 +886,18 @@ if hasattr(sslcontext_client, 'check_hostname'): sslcontext_client.check_hostname = True + # no CA loaded f_c = self.loop.create_connection(MyProto, host, port, ssl=sslcontext_client) - with test_utils.disable_logger(): - with self.assertRaisesRegex(ssl.SSLError, - 'certificate verify failed '): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.SSLError, + 'certificate verify failed '): + self.loop.run_until_complete(f_c) + + # execute the loop to log the connection error + test_utils.run_briefly(self.loop) # close connection self.assertIsNone(proto.transport) @@ -919,15 +924,20 @@ f_c = self.loop.create_unix_connection(MyProto, path, ssl=sslcontext_client, server_hostname='invalid') - with test_utils.disable_logger(): - with self.assertRaisesRegex(ssl.SSLError, - 'certificate verify failed '): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.SSLError, + 'certificate verify failed '): + self.loop.run_until_complete(f_c) + + # execute the loop to log the connection error + test_utils.run_briefly(self.loop) # close connection self.assertIsNone(proto.transport) server.close() + def test_legacy_create_unix_server_ssl_verify_failed(self): with test_utils.force_legacy_ssl_support(): self.test_create_unix_server_ssl_verify_failed() @@ -949,11 +959,12 @@ # incorrect server_hostname f_c = self.loop.create_connection(MyProto, host, port, ssl=sslcontext_client) - with test_utils.disable_logger(): - with self.assertRaisesRegex( - ssl.CertificateError, - "hostname '127.0.0.1' doesn't match 'localhost'"): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex( + ssl.CertificateError, + "hostname '127.0.0.1' doesn't match 'localhost'"): + self.loop.run_until_complete(f_c) # close connection proto.transport.close() diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -350,16 +350,13 @@ return transport def test_ctor(self): - tr = self.read_pipe_transport() + waiter = asyncio.Future(loop=self.loop) + tr = self.read_pipe_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.protocol.connection_made.assert_called_with(tr) self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.protocol.connection_made.assert_called_with(tr) - - def test_ctor_with_waiter(self): - fut = asyncio.Future(loop=self.loop) - tr = self.read_pipe_transport(waiter=fut) - test_utils.run_briefly(self.loop) - self.assertIsNone(fut.result()) + self.assertIsNone(waiter.result()) @mock.patch('os.read') def test__read_ready(self, m_read): @@ -502,17 +499,13 @@ return transport def test_ctor(self): - tr = self.write_pipe_transport() + waiter = asyncio.Future(loop=self.loop) + tr = self.write_pipe_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.protocol.connection_made.assert_called_with(tr) self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.protocol.connection_made.assert_called_with(tr) - - def test_ctor_with_waiter(self): - fut = asyncio.Future(loop=self.loop) - tr = self.write_pipe_transport(waiter=fut) - self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.assertEqual(None, fut.result()) + self.assertEqual(None, waiter.result()) def test_can_write_eof(self): tr = self.write_pipe_transport() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 14:35:14 2015 From: python-checkins at python.org (stefan.krah) Date: Thu, 29 Jan 2015 13:35:14 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Whitespace=2E?= Message-ID: <20150129133447.96078.18223@psf.io> https://hg.python.org/cpython/rev/7bb40e624387 changeset: 94374:7bb40e624387 user: Stefan Krah date: Thu Jan 29 14:33:37 2015 +0100 summary: Whitespace. files: Lib/test/test_memoryview.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -367,12 +367,12 @@ d = memoryview(b) del b - + self.assertEqual(c[0], 256) self.assertEqual(d[0], 256) self.assertEqual(c.format, "H") self.assertEqual(d.format, "H") - + _ = m.cast('I') self.assertEqual(c[0], 256) self.assertEqual(d[0], 256) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 14:35:14 2015 From: python-checkins at python.org (stefan.krah) Date: Thu, 29 Jan 2015 13:35:14 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNjY4?= =?utf-8?q?=3A_Ensure_that_format_strings_survive_slicing_after_casting=2E?= Message-ID: <20150129133447.25869.8347@psf.io> https://hg.python.org/cpython/rev/e9c1fca50b46 changeset: 94372:e9c1fca50b46 branch: 3.4 parent: 94370:c4fd6df9aea6 user: Stefan Krah date: Thu Jan 29 14:27:23 2015 +0100 summary: Issue #22668: Ensure that format strings survive slicing after casting. files: Include/memoryobject.h | 4 +- Lib/test/test_memoryview.py | 19 ++++++++ Objects/memoryobject.c | 56 +++++++++++++++++++++++- 3 files changed, 73 insertions(+), 6 deletions(-) diff --git a/Include/memoryobject.h b/Include/memoryobject.h --- a/Include/memoryobject.h +++ b/Include/memoryobject.h @@ -45,7 +45,7 @@ } _PyManagedBufferObject; -/* static storage used for casting between formats */ +/* deprecated, removed in 3.5 */ #define _Py_MEMORYVIEW_MAX_FORMAT 3 /* must be >= 3 */ /* memoryview state flags */ @@ -62,7 +62,7 @@ int flags; /* state flags */ Py_ssize_t exports; /* number of buffer re-exports */ Py_buffer view; /* private copy of the exporter's view */ - char format[_Py_MEMORYVIEW_MAX_FORMAT]; /* used for casting */ + char format[_Py_MEMORYVIEW_MAX_FORMAT]; /* deprecated, removed in 3.5 */ PyObject *weakreflist; Py_ssize_t ob_array[1]; /* shape, strides, suboffsets */ } PyMemoryViewObject; diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -360,6 +360,25 @@ self.assertEqual(list(reversed(m)), aslist) self.assertEqual(list(reversed(m)), list(m[::-1])) + def test_issue22668(self): + m = memoryview(bytes(range(8))) + b = m.cast('H') + c = b[0:2] + d = memoryview(b) + + del b + + self.assertEqual(c[0], 256) + self.assertEqual(d[0], 256) + self.assertEqual(c.format, "H") + self.assertEqual(d.format, "H") + + _ = m.cast('I') + self.assertEqual(c[0], 256) + self.assertEqual(d[0], 256) + self.assertEqual(c.format, "H") + self.assertEqual(d.format, "H") + # Variations on source objects for the buffer: bytes-like objects, then arrays # with itemsize > 1. diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -1135,6 +1135,51 @@ return -1; } +Py_LOCAL_INLINE(char *) +get_native_fmtstr(const char *fmt) +{ + int at = 0; + + if (fmt[0] == '@') { + at = 1; + fmt++; + } + if (fmt[0] == '\0' || fmt[1] != '\0') { + return NULL; + } + +#define RETURN(s) do { return at ? "@" s : s; } while (0) + + switch (fmt[0]) { + case 'c': RETURN("c"); + case 'b': RETURN("b"); + case 'B': RETURN("B"); + case 'h': RETURN("h"); + case 'H': RETURN("H"); + case 'i': RETURN("i"); + case 'I': RETURN("I"); + case 'l': RETURN("l"); + case 'L': RETURN("L"); + #ifdef HAVE_LONG_LONG + case 'q': RETURN("q"); + case 'Q': RETURN("Q"); + #endif + case 'n': RETURN("n"); + case 'N': RETURN("N"); + case 'f': RETURN("f"); + case 'd': RETURN("d"); + #ifdef HAVE_C99_BOOL + case '?': RETURN("?"); + #else + case '?': RETURN("?"); + #endif + case 'P': RETURN("P"); + } + + return NULL; +} + + /* Cast a memoryview's data type to 'format'. The input array must be C-contiguous. At least one of input-format, output-format must have byte size. The output array is 1-D, with the same byte length as the @@ -1184,10 +1229,13 @@ goto out; } - strncpy(mv->format, PyBytes_AS_STRING(asciifmt), - _Py_MEMORYVIEW_MAX_FORMAT); - mv->format[_Py_MEMORYVIEW_MAX_FORMAT-1] = '\0'; - view->format = mv->format; + view->format = get_native_fmtstr(PyBytes_AS_STRING(asciifmt)); + if (view->format == NULL) { + /* NOT_REACHED: get_native_fmtchar() already validates the format. */ + PyErr_SetString(PyExc_RuntimeError, + "memoryview: internal error"); + goto out; + } view->itemsize = itemsize; view->ndim = 1; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 14:35:14 2015 From: python-checkins at python.org (stefan.krah) Date: Thu, 29 Jan 2015 13:35:14 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogQ2xvc2VzICMyMjY2ODogTWVyZ2UgZnJvbSAzLjQu?= Message-ID: <20150129133447.96072.73816@psf.io> https://hg.python.org/cpython/rev/37112bd3dfb3 changeset: 94373:37112bd3dfb3 parent: 94371:497c9e57a54e parent: 94372:e9c1fca50b46 user: Stefan Krah date: Thu Jan 29 14:29:51 2015 +0100 summary: Closes #22668: Merge from 3.4. files: Doc/whatsnew/3.5.rst | 6 ++ Include/memoryobject.h | 4 - Lib/test/test_memoryview.py | 19 ++++++++ Lib/test/test_sys.py | 2 +- Objects/memoryobject.c | 56 +++++++++++++++++++++++- 5 files changed, 78 insertions(+), 9 deletions(-) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -485,6 +485,12 @@ Changes in the C API -------------------- +* The undocumented :c:member:`~PyMemoryViewObject.format` member of the + (non-public) :c:type:`PyMemoryViewObject` structure has been removed. + + All extensions relying on the relevant parts in ``memoryobject.h`` + must be rebuilt. + * The :c:type:`PyMemAllocator` structure was renamed to :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. diff --git a/Include/memoryobject.h b/Include/memoryobject.h --- a/Include/memoryobject.h +++ b/Include/memoryobject.h @@ -45,9 +45,6 @@ } _PyManagedBufferObject; -/* static storage used for casting between formats */ -#define _Py_MEMORYVIEW_MAX_FORMAT 3 /* must be >= 3 */ - /* memoryview state flags */ #define _Py_MEMORYVIEW_RELEASED 0x001 /* access to master buffer blocked */ #define _Py_MEMORYVIEW_C 0x002 /* C-contiguous layout */ @@ -62,7 +59,6 @@ int flags; /* state flags */ Py_ssize_t exports; /* number of buffer re-exports */ Py_buffer view; /* private copy of the exporter's view */ - char format[_Py_MEMORYVIEW_MAX_FORMAT]; /* used for casting */ PyObject *weakreflist; Py_ssize_t ob_array[1]; /* shape, strides, suboffsets */ } PyMemoryViewObject; diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -360,6 +360,25 @@ self.assertEqual(list(reversed(m)), aslist) self.assertEqual(list(reversed(m)), list(m[::-1])) + def test_issue22668(self): + m = memoryview(bytes(range(8))) + b = m.cast('H') + c = b[0:2] + d = memoryview(b) + + del b + + self.assertEqual(c[0], 256) + self.assertEqual(d[0], 256) + self.assertEqual(c.format, "H") + self.assertEqual(d.format, "H") + + _ = m.cast('I') + self.assertEqual(c[0], 256) + self.assertEqual(d[0], 256) + self.assertEqual(c.format, "H") + self.assertEqual(d.format, "H") + # Variations on source objects for the buffer: bytes-like objects, then arrays # with itemsize > 1. diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -965,7 +965,7 @@ check(int(PyLong_BASE**2-1), vsize('') + 2*self.longdigit) check(int(PyLong_BASE**2), vsize('') + 3*self.longdigit) # memoryview - check(memoryview(b''), size('Pnin 2P2n2i5P 3cPn')) + check(memoryview(b''), size('Pnin 2P2n2i5P Pn')) # module check(unittest, size('PnPPP')) # None diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -1132,6 +1132,51 @@ return -1; } +Py_LOCAL_INLINE(char *) +get_native_fmtstr(const char *fmt) +{ + int at = 0; + + if (fmt[0] == '@') { + at = 1; + fmt++; + } + if (fmt[0] == '\0' || fmt[1] != '\0') { + return NULL; + } + +#define RETURN(s) do { return at ? "@" s : s; } while (0) + + switch (fmt[0]) { + case 'c': RETURN("c"); + case 'b': RETURN("b"); + case 'B': RETURN("B"); + case 'h': RETURN("h"); + case 'H': RETURN("H"); + case 'i': RETURN("i"); + case 'I': RETURN("I"); + case 'l': RETURN("l"); + case 'L': RETURN("L"); + #ifdef HAVE_LONG_LONG + case 'q': RETURN("q"); + case 'Q': RETURN("Q"); + #endif + case 'n': RETURN("n"); + case 'N': RETURN("N"); + case 'f': RETURN("f"); + case 'd': RETURN("d"); + #ifdef HAVE_C99_BOOL + case '?': RETURN("?"); + #else + case '?': RETURN("?"); + #endif + case 'P': RETURN("P"); + } + + return NULL; +} + + /* Cast a memoryview's data type to 'format'. The input array must be C-contiguous. At least one of input-format, output-format must have byte size. The output array is 1-D, with the same byte length as the @@ -1181,10 +1226,13 @@ goto out; } - strncpy(mv->format, PyBytes_AS_STRING(asciifmt), - _Py_MEMORYVIEW_MAX_FORMAT); - mv->format[_Py_MEMORYVIEW_MAX_FORMAT-1] = '\0'; - view->format = mv->format; + view->format = get_native_fmtstr(PyBytes_AS_STRING(asciifmt)); + if (view->format == NULL) { + /* NOT_REACHED: get_native_fmtchar() already validates the format. */ + PyErr_SetString(PyExc_RuntimeError, + "memoryview: internal error"); + goto out; + } view->itemsize = itemsize; view->ndim = 1; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 17:41:54 2015 From: python-checkins at python.org (stefan.krah) Date: Thu, 29 Jan 2015 16:41:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322668=3A_Merge_from_3=2E4=2E?= Message-ID: <20150129164153.25869.53444@psf.io> https://hg.python.org/cpython/rev/da0ca7b1351f changeset: 94376:da0ca7b1351f parent: 94374:7bb40e624387 parent: 94375:9a4af12dcc9d user: Stefan Krah date: Thu Jan 29 17:40:59 2015 +0100 summary: Issue #22668: Merge from 3.4. files: Lib/test/test_memoryview.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -361,7 +361,9 @@ self.assertEqual(list(reversed(m)), list(m[::-1])) def test_issue22668(self): - m = memoryview(bytes(range(8))) + a = array.array('H', [256, 256, 256, 256]) + x = memoryview(a) + m = x.cast('B') b = m.cast('H') c = b[0:2] d = memoryview(b) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 17:41:54 2015 From: python-checkins at python.org (stefan.krah) Date: Thu, 29 Jan 2015 16:41:54 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNjY4?= =?utf-8?q?=3A_Remove_endianness_assumption_in_test=2E?= Message-ID: <20150129164153.25845.36191@psf.io> https://hg.python.org/cpython/rev/9a4af12dcc9d changeset: 94375:9a4af12dcc9d branch: 3.4 parent: 94372:e9c1fca50b46 user: Stefan Krah date: Thu Jan 29 17:33:31 2015 +0100 summary: Issue #22668: Remove endianness assumption in test. files: Lib/test/test_memoryview.py | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -361,18 +361,20 @@ self.assertEqual(list(reversed(m)), list(m[::-1])) def test_issue22668(self): - m = memoryview(bytes(range(8))) + a = array.array('H', [256, 256, 256, 256]) + x = memoryview(a) + m = x.cast('B') b = m.cast('H') c = b[0:2] d = memoryview(b) del b - + self.assertEqual(c[0], 256) self.assertEqual(d[0], 256) self.assertEqual(c.format, "H") self.assertEqual(d.format, "H") - + _ = m.cast('I') self.assertEqual(c[0], 256) self.assertEqual(d[0], 256) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Jan 29 17:54:06 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 16:54:06 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjQz?= =?utf-8?q?=2C_asyncio=3A_Emit_a_ResourceWarning_when_an_event_loop_or_a_t?= =?utf-8?q?ransport?= Message-ID: <20150129165357.25853.70559@psf.io> https://hg.python.org/cpython/rev/543f770f62f0 changeset: 94377:543f770f62f0 branch: 3.4 parent: 94375:9a4af12dcc9d user: Victor Stinner date: Thu Jan 29 17:50:58 2015 +0100 summary: Issue #23243, asyncio: Emit a ResourceWarning when an event loop or a transport is not explicitly closed. Close also explicitly transports in test_sslproto. files: Lib/asyncio/base_events.py | 11 +++++ Lib/asyncio/base_subprocess.py | 19 +++++++++- Lib/asyncio/futures.py | 6 +- Lib/asyncio/proactor_events.py | 11 +++++ Lib/asyncio/selector_events.py | 16 ++++++++ Lib/asyncio/sslproto.py | 13 ++++++ Lib/asyncio/unix_events.py | 19 ++++++++++ Lib/asyncio/windows_utils.py | 6 ++- Lib/test/test_asyncio/test_proactor_events.py | 6 ++- Lib/test/test_asyncio/test_sslproto.py | 7 +-- 10 files changed, 104 insertions(+), 10 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -26,6 +26,7 @@ import time import traceback import sys +import warnings from . import coroutines from . import events @@ -333,6 +334,16 @@ """Returns True if the event loop was closed.""" return self._closed + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self.is_closed(): + warnings.warn("unclosed event loop %r" % self, ResourceWarning) + if not self.is_running(): + self.close() + def is_running(self): """Returns True if the event loop is running.""" return (self._owner is not None) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -1,5 +1,7 @@ import collections import subprocess +import sys +import warnings from . import protocols from . import transports @@ -13,6 +15,7 @@ stdin, stdout, stderr, bufsize, extra=None, **kwargs): super().__init__(extra) + self._closed = False self._protocol = protocol self._loop = loop self._pid = None @@ -40,7 +43,10 @@ program, self._pid) def __repr__(self): - info = [self.__class__.__name__, 'pid=%s' % self._pid] + info = [self.__class__.__name__] + if self._closed: + info.append('closed') + info.append('pid=%s' % self._pid) if self._returncode is not None: info.append('returncode=%s' % self._returncode) @@ -70,6 +76,7 @@ raise NotImplementedError def close(self): + self._closed = True for proto in self._pipes.values(): if proto is None: continue @@ -77,6 +84,15 @@ if self._returncode is None: self.terminate() + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self._closed: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() + def get_pid(self): return self._pid @@ -104,6 +120,7 @@ Function called when an exception is raised during the creation of a subprocess. """ + self._closed = True if self._loop.get_debug(): logger.warning('Exception during subprocess creation, ' 'kill the subprocess %r', diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -195,9 +195,9 @@ info = self._repr_info() return '<%s %s>' % (self.__class__.__name__, ' '.join(info)) - # On Python 3.3 or older, objects with a destructor part of a reference - # cycle are never destroyed. It's not more the case on Python 3.4 thanks to - # the PEP 442. + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. if _PY34: def __del__(self): if not self._log_traceback: diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -7,6 +7,8 @@ __all__ = ['BaseProactorEventLoop'] import socket +import sys +import warnings from . import base_events from . import constants @@ -74,6 +76,15 @@ self._read_fut.cancel() self._read_fut = None + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._sock is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() + def _fatal_error(self, exc, message='Fatal error on pipe transport'): if isinstance(exc, (BrokenPipeError, ConnectionResetError)): if self._loop.get_debug(): diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -10,6 +10,8 @@ import errno import functools import socket +import sys +import warnings try: import ssl except ImportError: # pragma: no cover @@ -499,6 +501,11 @@ _buffer_factory = bytearray # Constructs initial value for self._buffer. + # Attribute used in the destructor: it must be set even if the constructor + # is not called (see _SelectorSslTransport which may start by raising an + # exception) + _sock = None + def __init__(self, loop, sock, protocol, extra=None, server=None): super().__init__(extra, loop) self._extra['socket'] = sock @@ -559,6 +566,15 @@ self._conn_lost += 1 self._loop.call_soon(self._call_connection_lost, None) + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._sock is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._sock.close() + def _fatal_error(self, exc, message='Fatal error on transport'): # Should be called from exception handler only. if isinstance(exc, (BrokenPipeError, diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -1,4 +1,6 @@ import collections +import sys +import warnings try: import ssl except ImportError: # pragma: no cover @@ -295,6 +297,7 @@ self._loop = loop self._ssl_protocol = ssl_protocol self._app_protocol = app_protocol + self._closed = False def get_extra_info(self, name, default=None): """Get optional transport information.""" @@ -308,8 +311,18 @@ protocol's connection_lost() method will (eventually) called with None as its argument. """ + self._closed = True self._ssl_protocol._start_shutdown() + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self._closed: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() + def pause_reading(self): """Pause the receiving end. diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -8,6 +8,7 @@ import subprocess import sys import threading +import warnings from . import base_events @@ -353,6 +354,15 @@ if not self._closing: self._close(None) + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._pipe is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._pipe.close() + def _fatal_error(self, exc, message='Fatal error on pipe transport'): # should be called by exception handler only if (isinstance(exc, OSError) and exc.errno == errno.EIO): @@ -529,6 +539,15 @@ # write_eof is all what we needed to close the write pipe self.write_eof() + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._pipe is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._pipe.close() + def abort(self): self._close(None) diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -14,6 +14,7 @@ import socket import subprocess import tempfile +import warnings __all__ = ['socketpair', 'pipe', 'Popen', 'PIPE', 'PipeHandle'] @@ -156,7 +157,10 @@ CloseHandle(self._handle) self._handle = None - __del__ = close + def __del__(self): + if self._handle is not None: + warnings.warn("unclosed %r" % self, ResourceWarning) + self.close() def __enter__(self): return self diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -499,8 +499,12 @@ self.proactor.accept.assert_called_with(self.sock) def test_socketpair(self): + class EventLoop(BaseProactorEventLoop): + # override the destructor to not log a ResourceWarning + def __del__(self): + pass self.assertRaises( - NotImplementedError, BaseProactorEventLoop, self.proactor) + NotImplementedError, EventLoop, self.proactor) def test_make_socket_transport(self): tr = self.loop._make_socket_transport(self.sock, asyncio.Protocol()) diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -22,7 +22,9 @@ def ssl_protocol(self, waiter=None): sslcontext = test_utils.dummy_ssl_context() app_proto = asyncio.Protocol() - return sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter) + proto = sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter) + self.addCleanup(proto._app_transport.close) + return proto def connection_made(self, ssl_proto, do_handshake=None): transport = mock.Mock() @@ -56,9 +58,6 @@ with test_utils.disable_logger(): self.loop.run_until_complete(handshake_fut) - # Close the transport - ssl_proto._app_transport.close() - def test_eof_received_waiter(self): waiter = asyncio.Future(loop=self.loop) ssl_proto = self.ssl_protocol(waiter) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:09:05 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:09:05 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150129230858.25871.63719@psf.io> https://hg.python.org/cpython/rev/7c9a42cbfff0 changeset: 94378:7c9a42cbfff0 parent: 94376:da0ca7b1351f parent: 94377:543f770f62f0 user: Victor Stinner date: Fri Jan 30 00:04:27 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 11 +++++ Lib/asyncio/base_subprocess.py | 19 +++++++++- Lib/asyncio/futures.py | 6 +- Lib/asyncio/proactor_events.py | 11 +++++ Lib/asyncio/selector_events.py | 16 ++++++++ Lib/asyncio/sslproto.py | 13 ++++++ Lib/asyncio/unix_events.py | 19 ++++++++++ Lib/asyncio/windows_utils.py | 6 ++- Lib/test/test_asyncio/test_proactor_events.py | 6 ++- Lib/test/test_asyncio/test_sslproto.py | 7 +-- 10 files changed, 104 insertions(+), 10 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -26,6 +26,7 @@ import time import traceback import sys +import warnings from . import coroutines from . import events @@ -333,6 +334,16 @@ """Returns True if the event loop was closed.""" return self._closed + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self.is_closed(): + warnings.warn("unclosed event loop %r" % self, ResourceWarning) + if not self.is_running(): + self.close() + def is_running(self): """Returns True if the event loop is running.""" return (self._owner is not None) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -1,5 +1,7 @@ import collections import subprocess +import sys +import warnings from . import protocols from . import transports @@ -13,6 +15,7 @@ stdin, stdout, stderr, bufsize, extra=None, **kwargs): super().__init__(extra) + self._closed = False self._protocol = protocol self._loop = loop self._pid = None @@ -40,7 +43,10 @@ program, self._pid) def __repr__(self): - info = [self.__class__.__name__, 'pid=%s' % self._pid] + info = [self.__class__.__name__] + if self._closed: + info.append('closed') + info.append('pid=%s' % self._pid) if self._returncode is not None: info.append('returncode=%s' % self._returncode) @@ -70,6 +76,7 @@ raise NotImplementedError def close(self): + self._closed = True for proto in self._pipes.values(): if proto is None: continue @@ -77,6 +84,15 @@ if self._returncode is None: self.terminate() + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self._closed: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() + def get_pid(self): return self._pid @@ -104,6 +120,7 @@ Function called when an exception is raised during the creation of a subprocess. """ + self._closed = True if self._loop.get_debug(): logger.warning('Exception during subprocess creation, ' 'kill the subprocess %r', diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -195,9 +195,9 @@ info = self._repr_info() return '<%s %s>' % (self.__class__.__name__, ' '.join(info)) - # On Python 3.3 or older, objects with a destructor part of a reference - # cycle are never destroyed. It's not more the case on Python 3.4 thanks to - # the PEP 442. + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. if _PY34: def __del__(self): if not self._log_traceback: diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -7,6 +7,8 @@ __all__ = ['BaseProactorEventLoop'] import socket +import sys +import warnings from . import base_events from . import constants @@ -74,6 +76,15 @@ self._read_fut.cancel() self._read_fut = None + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._sock is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() + def _fatal_error(self, exc, message='Fatal error on pipe transport'): if isinstance(exc, (BrokenPipeError, ConnectionResetError)): if self._loop.get_debug(): diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -10,6 +10,8 @@ import errno import functools import socket +import sys +import warnings try: import ssl except ImportError: # pragma: no cover @@ -499,6 +501,11 @@ _buffer_factory = bytearray # Constructs initial value for self._buffer. + # Attribute used in the destructor: it must be set even if the constructor + # is not called (see _SelectorSslTransport which may start by raising an + # exception) + _sock = None + def __init__(self, loop, sock, protocol, extra=None, server=None): super().__init__(extra, loop) self._extra['socket'] = sock @@ -559,6 +566,15 @@ self._conn_lost += 1 self._loop.call_soon(self._call_connection_lost, None) + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._sock is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._sock.close() + def _fatal_error(self, exc, message='Fatal error on transport'): # Should be called from exception handler only. if isinstance(exc, (BrokenPipeError, diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -1,4 +1,6 @@ import collections +import sys +import warnings try: import ssl except ImportError: # pragma: no cover @@ -295,6 +297,7 @@ self._loop = loop self._ssl_protocol = ssl_protocol self._app_protocol = app_protocol + self._closed = False def get_extra_info(self, name, default=None): """Get optional transport information.""" @@ -308,8 +311,18 @@ protocol's connection_lost() method will (eventually) called with None as its argument. """ + self._closed = True self._ssl_protocol._start_shutdown() + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self._closed: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() + def pause_reading(self): """Pause the receiving end. diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -8,6 +8,7 @@ import subprocess import sys import threading +import warnings from . import base_events @@ -353,6 +354,15 @@ if not self._closing: self._close(None) + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._pipe is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._pipe.close() + def _fatal_error(self, exc, message='Fatal error on pipe transport'): # should be called by exception handler only if (isinstance(exc, OSError) and exc.errno == errno.EIO): @@ -529,6 +539,15 @@ # write_eof is all what we needed to close the write pipe self.write_eof() + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._pipe is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._pipe.close() + def abort(self): self._close(None) diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -14,6 +14,7 @@ import socket import subprocess import tempfile +import warnings __all__ = ['socketpair', 'pipe', 'Popen', 'PIPE', 'PipeHandle'] @@ -156,7 +157,10 @@ CloseHandle(self._handle) self._handle = None - __del__ = close + def __del__(self): + if self._handle is not None: + warnings.warn("unclosed %r" % self, ResourceWarning) + self.close() def __enter__(self): return self diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -499,8 +499,12 @@ self.proactor.accept.assert_called_with(self.sock) def test_socketpair(self): + class EventLoop(BaseProactorEventLoop): + # override the destructor to not log a ResourceWarning + def __del__(self): + pass self.assertRaises( - NotImplementedError, BaseProactorEventLoop, self.proactor) + NotImplementedError, EventLoop, self.proactor) def test_make_socket_transport(self): tr = self.loop._make_socket_transport(self.sock, asyncio.Protocol()) diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -22,7 +22,9 @@ def ssl_protocol(self, waiter=None): sslcontext = test_utils.dummy_ssl_context() app_proto = asyncio.Protocol() - return sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter) + proto = sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter) + self.addCleanup(proto._app_transport.close) + return proto def connection_made(self, ssl_proto, do_handshake=None): transport = mock.Mock() @@ -56,9 +58,6 @@ with test_utils.disable_logger(): self.loop.run_until_complete(handshake_fut) - # Close the transport - ssl_proto._app_transport.close() - def test_eof_received_waiter(self): waiter = asyncio.Future(loop=self.loop) ssl_proto = self.ssl_protocol(waiter) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:09:05 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:09:05 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150129230859.39272.82935@psf.io> https://hg.python.org/cpython/rev/0701ffaa89f9 changeset: 94380:0701ffaa89f9 parent: 94378:7c9a42cbfff0 parent: 94379:a5efd5021ca1 user: Victor Stinner date: Fri Jan 30 00:05:36 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_subprocess.py | 108 ++++++---- Lib/asyncio/subprocess.py | 40 +--- Lib/asyncio/unix_events.py | 15 +- Lib/asyncio/windows_events.py | 7 +- Lib/test/test_asyncio/test_events.py | 35 +- Lib/test/test_asyncio/test_subprocess.py | 65 ++++++ 6 files changed, 166 insertions(+), 104 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -3,6 +3,7 @@ import sys import warnings +from . import futures from . import protocols from . import transports from .coroutines import coroutine @@ -13,27 +14,32 @@ def __init__(self, loop, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=None, **kwargs): + waiter=None, extra=None, **kwargs): super().__init__(extra) self._closed = False self._protocol = protocol self._loop = loop + self._proc = None self._pid = None + self._returncode = None + self._exit_waiters = [] + self._pending_calls = collections.deque() + self._pipes = {} + self._finished = False - self._pipes = {} if stdin == subprocess.PIPE: self._pipes[0] = None if stdout == subprocess.PIPE: self._pipes[1] = None if stderr == subprocess.PIPE: self._pipes[2] = None - self._pending_calls = collections.deque() - self._finished = False - self._returncode = None + + # Create the child process: set the _proc attribute self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, stderr=stderr, bufsize=bufsize, **kwargs) self._pid = self._proc.pid self._extra['subprocess'] = self._proc + if self._loop.get_debug(): if isinstance(args, (bytes, str)): program = args @@ -42,6 +48,8 @@ logger.debug('process %r created: pid %s', program, self._pid) + self._loop.create_task(self._connect_pipes(waiter)) + def __repr__(self): info = [self.__class__.__name__] if self._closed: @@ -77,12 +85,23 @@ def close(self): self._closed = True + for proto in self._pipes.values(): if proto is None: continue proto.pipe.close() - if self._returncode is None: - self.terminate() + + if self._proc is not None and self._returncode is None: + if self._loop.get_debug(): + logger.warning('Close running child process: kill %r', self) + + try: + self._proc.kill() + except ProcessLookupError: + pass + + # Don't clear the _proc reference yet because _post_init() may + # still run # On Python 3.3 and older, objects with a destructor part of a reference # cycle are never destroyed. It's not more the case on Python 3.4 thanks @@ -105,59 +124,42 @@ else: return None + def _check_proc(self): + if self._closed: + raise ValueError("operation on closed transport") + if self._proc is None: + raise ProcessLookupError() + def send_signal(self, signal): + self._check_proc() self._proc.send_signal(signal) def terminate(self): + self._check_proc() self._proc.terminate() def kill(self): + self._check_proc() self._proc.kill() - def _kill_wait(self): - """Close pipes, kill the subprocess and read its return status. - - Function called when an exception is raised during the creation - of a subprocess. - """ - self._closed = True - if self._loop.get_debug(): - logger.warning('Exception during subprocess creation, ' - 'kill the subprocess %r', - self, - exc_info=True) - - proc = self._proc - if proc.stdout: - proc.stdout.close() - if proc.stderr: - proc.stderr.close() - if proc.stdin: - proc.stdin.close() - - try: - proc.kill() - except ProcessLookupError: - pass - self._returncode = proc.wait() - - self.close() - @coroutine - def _post_init(self): + def _connect_pipes(self, waiter): try: proc = self._proc loop = self._loop + if proc.stdin is not None: _, pipe = yield from loop.connect_write_pipe( lambda: WriteSubprocessPipeProto(self, 0), proc.stdin) self._pipes[0] = pipe + if proc.stdout is not None: _, pipe = yield from loop.connect_read_pipe( lambda: ReadSubprocessPipeProto(self, 1), proc.stdout) self._pipes[1] = pipe + if proc.stderr is not None: _, pipe = yield from loop.connect_read_pipe( lambda: ReadSubprocessPipeProto(self, 2), @@ -166,13 +168,16 @@ assert self._pending_calls is not None - self._loop.call_soon(self._protocol.connection_made, self) + loop.call_soon(self._protocol.connection_made, self) for callback, data in self._pending_calls: - self._loop.call_soon(callback, *data) + loop.call_soon(callback, *data) self._pending_calls = None - except: - self._kill_wait() - raise + except Exception as exc: + if waiter is not None and not waiter.cancelled(): + waiter.set_exception(exc) + else: + if waiter is not None and not waiter.cancelled(): + waiter.set_result(None) def _call(self, cb, *data): if self._pending_calls is not None: @@ -197,6 +202,23 @@ self._call(self._protocol.process_exited) self._try_finish() + # wake up futures waiting for wait() + for waiter in self._exit_waiters: + if not waiter.cancelled(): + waiter.set_result(returncode) + self._exit_waiters = None + + def wait(self): + """Wait until the process exit and return the process return code. + + This method is a coroutine.""" + if self._returncode is not None: + return self._returncode + + waiter = futures.Future(loop=self._loop) + self._exit_waiters.append(waiter) + return (yield from waiter) + def _try_finish(self): assert not self._finished if self._returncode is None: @@ -210,9 +232,9 @@ try: self._protocol.connection_lost(exc) finally: + self._loop = None self._proc = None self._protocol = None - self._loop = None class WriteSubprocessPipeProto(protocols.BaseProtocol): diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -25,8 +25,6 @@ super().__init__(loop=loop) self._limit = limit self.stdin = self.stdout = self.stderr = None - self.waiter = futures.Future(loop=loop) - self._waiters = collections.deque() self._transport = None def __repr__(self): @@ -61,9 +59,6 @@ reader=None, loop=self._loop) - if not self.waiter.cancelled(): - self.waiter.set_result(None) - def pipe_data_received(self, fd, data): if fd == 1: reader = self.stdout @@ -94,16 +89,9 @@ reader.set_exception(exc) def process_exited(self): - returncode = self._transport.get_returncode() self._transport.close() self._transport = None - # wake up futures waiting for wait() - while self._waiters: - waiter = self._waiters.popleft() - if not waiter.cancelled(): - waiter.set_result(returncode) - class Process: def __init__(self, transport, protocol, loop): @@ -124,30 +112,18 @@ @coroutine def wait(self): - """Wait until the process exit and return the process return code.""" - returncode = self._transport.get_returncode() - if returncode is not None: - return returncode + """Wait until the process exit and return the process return code. - waiter = futures.Future(loop=self._loop) - self._protocol._waiters.append(waiter) - yield from waiter - return waiter.result() - - def _check_alive(self): - if self._transport.get_returncode() is not None: - raise ProcessLookupError() + This method is a coroutine.""" + return (yield from self._transport.wait()) def send_signal(self, signal): - self._check_alive() self._transport.send_signal(signal) def terminate(self): - self._check_alive() self._transport.terminate() def kill(self): - self._check_alive() self._transport.kill() @coroutine @@ -221,11 +197,6 @@ protocol_factory, cmd, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - try: - yield from protocol.waiter - except: - transport._kill_wait() - raise return Process(transport, protocol, loop) @coroutine @@ -241,9 +212,4 @@ program, *args, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - try: - yield from protocol.waiter - except: - transport._kill_wait() - raise return Process(transport, protocol, loop) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -16,6 +16,7 @@ from . import constants from . import coroutines from . import events +from . import futures from . import selector_events from . import selectors from . import transports @@ -175,16 +176,20 @@ stdin, stdout, stderr, bufsize, extra=None, **kwargs): with events.get_child_watcher() as watcher: + waiter = futures.Future(loop=self) transp = _UnixSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=extra, **kwargs) + waiter=waiter, extra=extra, + **kwargs) + + watcher.add_child_handler(transp.get_pid(), + self._child_watcher_callback, transp) try: - yield from transp._post_init() + yield from waiter except: transp.close() + yield from transp.wait() raise - watcher.add_child_handler(transp.get_pid(), - self._child_watcher_callback, transp) return transp @@ -774,7 +779,7 @@ pass def add_child_handler(self, pid, callback, *args): - self._callbacks[pid] = callback, args + self._callbacks[pid] = (callback, args) # Prevent a race condition in case the child is already terminated. self._do_waitpid(pid) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -366,13 +366,16 @@ def _make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra=None, **kwargs): + waiter = futures.Future(loop=self) transp = _WindowsSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=extra, **kwargs) + waiter=waiter, extra=extra, + **kwargs) try: - yield from transp._post_init() + yield from waiter except: transp.close() + yield from transp.wait() raise return transp diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1551,9 +1551,10 @@ stdin = transp.get_pipe_transport(0) stdin.write(b'Python The Winner') self.loop.run_until_complete(proto.got_data[1].wait()) - transp.close() + with test_utils.disable_logger(): + transp.close() self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) self.assertEqual(b'Python The Winner', proto.data[1]) def test_subprocess_interactive(self): @@ -1567,21 +1568,20 @@ self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - try: - stdin = transp.get_pipe_transport(0) - stdin.write(b'Python ') - self.loop.run_until_complete(proto.got_data[1].wait()) - proto.got_data[1].clear() - self.assertEqual(b'Python ', proto.data[1]) - - stdin.write(b'The Winner') - self.loop.run_until_complete(proto.got_data[1].wait()) - self.assertEqual(b'Python The Winner', proto.data[1]) - finally: + stdin = transp.get_pipe_transport(0) + stdin.write(b'Python ') + self.loop.run_until_complete(proto.got_data[1].wait()) + proto.got_data[1].clear() + self.assertEqual(b'Python ', proto.data[1]) + + stdin.write(b'The Winner') + self.loop.run_until_complete(proto.got_data[1].wait()) + self.assertEqual(b'Python The Winner', proto.data[1]) + + with test_utils.disable_logger(): transp.close() - self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) def test_subprocess_shell(self): connect = self.loop.subprocess_shell( @@ -1739,9 +1739,10 @@ # GetLastError()==ERROR_INVALID_NAME on Windows!?! (Using # WriteFile() we get ERROR_BROKEN_PIPE as expected.) self.assertEqual(b'ERR:OSError', proto.data[2]) - transp.close() + with test_utils.disable_logger(): + transp.close() self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) def test_subprocess_wait_no_same_group(self): # start the new process in a new session diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -4,6 +4,7 @@ from unittest import mock import asyncio +from asyncio import base_subprocess from asyncio import subprocess from asyncio import test_utils try: @@ -23,6 +24,70 @@ 'data = sys.stdin.buffer.read()', 'sys.stdout.buffer.write(data)'))] +class TestSubprocessTransport(base_subprocess.BaseSubprocessTransport): + def _start(self, *args, **kwargs): + self._proc = mock.Mock() + self._proc.stdin = None + self._proc.stdout = None + self._proc.stderr = None + + +class SubprocessTransportTests(test_utils.TestCase): + def setUp(self): + self.loop = self.new_test_loop() + self.set_event_loop(self.loop) + + + def create_transport(self, waiter=None): + protocol = mock.Mock() + protocol.connection_made._is_coroutine = False + protocol.process_exited._is_coroutine = False + transport = TestSubprocessTransport( + self.loop, protocol, ['test'], False, + None, None, None, 0, waiter=waiter) + return (transport, protocol) + + def test_close(self): + waiter = asyncio.Future(loop=self.loop) + transport, protocol = self.create_transport(waiter) + transport._process_exited(0) + transport.close() + + # The loop didn't run yet + self.assertFalse(protocol.connection_made.called) + + # methods must raise ProcessLookupError if the transport was closed + self.assertRaises(ValueError, transport.send_signal, signal.SIGTERM) + self.assertRaises(ValueError, transport.terminate) + self.assertRaises(ValueError, transport.kill) + + self.loop.run_until_complete(waiter) + + def test_proc_exited(self): + waiter = asyncio.Future(loop=self.loop) + transport, protocol = self.create_transport(waiter) + transport._process_exited(6) + self.loop.run_until_complete(waiter) + + self.assertEqual(transport.get_returncode(), 6) + + self.assertTrue(protocol.connection_made.called) + self.assertTrue(protocol.process_exited.called) + self.assertTrue(protocol.connection_lost.called) + self.assertEqual(protocol.connection_lost.call_args[0], (None,)) + + self.assertFalse(transport._closed) + self.assertIsNone(transport._loop) + self.assertIsNone(transport._proc) + self.assertIsNone(transport._protocol) + + # methods must raise ProcessLookupError if the process exited + self.assertRaises(ProcessLookupError, + transport.send_signal, signal.SIGTERM) + self.assertRaises(ProcessLookupError, transport.terminate) + self.assertRaises(ProcessLookupError, transport.kill) + + class SubprocessMixin: def test_stdin_stdout(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:09:05 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:09:05 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogc3lu?= =?utf-8?q?c_with_Tulip?= Message-ID: <20150129230859.106370.28895@psf.io> https://hg.python.org/cpython/rev/a5efd5021ca1 changeset: 94379:a5efd5021ca1 branch: 3.4 parent: 94377:543f770f62f0 user: Victor Stinner date: Fri Jan 30 00:05:19 2015 +0100 summary: asyncio: sync with Tulip Issue #23347: send_signal(), kill() and terminate() methods of BaseSubprocessTransport now check if the transport was closed and if the process exited. Issue #23347: Refactor creation of subprocess transports. Changes on BaseSubprocessTransport: * Add a wait() method to wait until the child process exit * The constructor now accepts an optional waiter parameter. The _post_init() coroutine must not be called explicitly anymore. It makes subprocess transports closer to other transports, and it gives more freedom if we want later to change completly how subprocess transports are created. * close() now kills the process instead of kindly terminate it: the child process may ignore SIGTERM and continue to run. Call explicitly terminate() and wait() if you want to kindly terminate the child process. * close() now logs a warning in debug mode if the process is still running and needs to be killed * _make_subprocess_transport() is now fully asynchronous again: if the creation of the transport failed, wait asynchronously for the process eixt. Before the wait was synchronous. This change requires close() to *kill*, and not terminate, the child process. * Remove the _kill_wait() method, replaced with a more agressive close() method. It fixes _make_subprocess_transport() on error. BaseSubprocessTransport.close() calls the close() method of pipe transports, whereas _kill_wait() closed directly pipes of the subprocess.Popen object without unregistering file descriptors from the selector (which caused severe bugs). These changes simplifies the code of subprocess.py. files: Lib/asyncio/base_subprocess.py | 108 ++++++---- Lib/asyncio/subprocess.py | 40 +--- Lib/asyncio/unix_events.py | 15 +- Lib/asyncio/windows_events.py | 7 +- Lib/test/test_asyncio/test_events.py | 35 +- Lib/test/test_asyncio/test_subprocess.py | 65 ++++++ 6 files changed, 166 insertions(+), 104 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -3,6 +3,7 @@ import sys import warnings +from . import futures from . import protocols from . import transports from .coroutines import coroutine @@ -13,27 +14,32 @@ def __init__(self, loop, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=None, **kwargs): + waiter=None, extra=None, **kwargs): super().__init__(extra) self._closed = False self._protocol = protocol self._loop = loop + self._proc = None self._pid = None + self._returncode = None + self._exit_waiters = [] + self._pending_calls = collections.deque() + self._pipes = {} + self._finished = False - self._pipes = {} if stdin == subprocess.PIPE: self._pipes[0] = None if stdout == subprocess.PIPE: self._pipes[1] = None if stderr == subprocess.PIPE: self._pipes[2] = None - self._pending_calls = collections.deque() - self._finished = False - self._returncode = None + + # Create the child process: set the _proc attribute self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, stderr=stderr, bufsize=bufsize, **kwargs) self._pid = self._proc.pid self._extra['subprocess'] = self._proc + if self._loop.get_debug(): if isinstance(args, (bytes, str)): program = args @@ -42,6 +48,8 @@ logger.debug('process %r created: pid %s', program, self._pid) + self._loop.create_task(self._connect_pipes(waiter)) + def __repr__(self): info = [self.__class__.__name__] if self._closed: @@ -77,12 +85,23 @@ def close(self): self._closed = True + for proto in self._pipes.values(): if proto is None: continue proto.pipe.close() - if self._returncode is None: - self.terminate() + + if self._proc is not None and self._returncode is None: + if self._loop.get_debug(): + logger.warning('Close running child process: kill %r', self) + + try: + self._proc.kill() + except ProcessLookupError: + pass + + # Don't clear the _proc reference yet because _post_init() may + # still run # On Python 3.3 and older, objects with a destructor part of a reference # cycle are never destroyed. It's not more the case on Python 3.4 thanks @@ -105,59 +124,42 @@ else: return None + def _check_proc(self): + if self._closed: + raise ValueError("operation on closed transport") + if self._proc is None: + raise ProcessLookupError() + def send_signal(self, signal): + self._check_proc() self._proc.send_signal(signal) def terminate(self): + self._check_proc() self._proc.terminate() def kill(self): + self._check_proc() self._proc.kill() - def _kill_wait(self): - """Close pipes, kill the subprocess and read its return status. - - Function called when an exception is raised during the creation - of a subprocess. - """ - self._closed = True - if self._loop.get_debug(): - logger.warning('Exception during subprocess creation, ' - 'kill the subprocess %r', - self, - exc_info=True) - - proc = self._proc - if proc.stdout: - proc.stdout.close() - if proc.stderr: - proc.stderr.close() - if proc.stdin: - proc.stdin.close() - - try: - proc.kill() - except ProcessLookupError: - pass - self._returncode = proc.wait() - - self.close() - @coroutine - def _post_init(self): + def _connect_pipes(self, waiter): try: proc = self._proc loop = self._loop + if proc.stdin is not None: _, pipe = yield from loop.connect_write_pipe( lambda: WriteSubprocessPipeProto(self, 0), proc.stdin) self._pipes[0] = pipe + if proc.stdout is not None: _, pipe = yield from loop.connect_read_pipe( lambda: ReadSubprocessPipeProto(self, 1), proc.stdout) self._pipes[1] = pipe + if proc.stderr is not None: _, pipe = yield from loop.connect_read_pipe( lambda: ReadSubprocessPipeProto(self, 2), @@ -166,13 +168,16 @@ assert self._pending_calls is not None - self._loop.call_soon(self._protocol.connection_made, self) + loop.call_soon(self._protocol.connection_made, self) for callback, data in self._pending_calls: - self._loop.call_soon(callback, *data) + loop.call_soon(callback, *data) self._pending_calls = None - except: - self._kill_wait() - raise + except Exception as exc: + if waiter is not None and not waiter.cancelled(): + waiter.set_exception(exc) + else: + if waiter is not None and not waiter.cancelled(): + waiter.set_result(None) def _call(self, cb, *data): if self._pending_calls is not None: @@ -197,6 +202,23 @@ self._call(self._protocol.process_exited) self._try_finish() + # wake up futures waiting for wait() + for waiter in self._exit_waiters: + if not waiter.cancelled(): + waiter.set_result(returncode) + self._exit_waiters = None + + def wait(self): + """Wait until the process exit and return the process return code. + + This method is a coroutine.""" + if self._returncode is not None: + return self._returncode + + waiter = futures.Future(loop=self._loop) + self._exit_waiters.append(waiter) + return (yield from waiter) + def _try_finish(self): assert not self._finished if self._returncode is None: @@ -210,9 +232,9 @@ try: self._protocol.connection_lost(exc) finally: + self._loop = None self._proc = None self._protocol = None - self._loop = None class WriteSubprocessPipeProto(protocols.BaseProtocol): diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -25,8 +25,6 @@ super().__init__(loop=loop) self._limit = limit self.stdin = self.stdout = self.stderr = None - self.waiter = futures.Future(loop=loop) - self._waiters = collections.deque() self._transport = None def __repr__(self): @@ -61,9 +59,6 @@ reader=None, loop=self._loop) - if not self.waiter.cancelled(): - self.waiter.set_result(None) - def pipe_data_received(self, fd, data): if fd == 1: reader = self.stdout @@ -94,16 +89,9 @@ reader.set_exception(exc) def process_exited(self): - returncode = self._transport.get_returncode() self._transport.close() self._transport = None - # wake up futures waiting for wait() - while self._waiters: - waiter = self._waiters.popleft() - if not waiter.cancelled(): - waiter.set_result(returncode) - class Process: def __init__(self, transport, protocol, loop): @@ -124,30 +112,18 @@ @coroutine def wait(self): - """Wait until the process exit and return the process return code.""" - returncode = self._transport.get_returncode() - if returncode is not None: - return returncode + """Wait until the process exit and return the process return code. - waiter = futures.Future(loop=self._loop) - self._protocol._waiters.append(waiter) - yield from waiter - return waiter.result() - - def _check_alive(self): - if self._transport.get_returncode() is not None: - raise ProcessLookupError() + This method is a coroutine.""" + return (yield from self._transport.wait()) def send_signal(self, signal): - self._check_alive() self._transport.send_signal(signal) def terminate(self): - self._check_alive() self._transport.terminate() def kill(self): - self._check_alive() self._transport.kill() @coroutine @@ -221,11 +197,6 @@ protocol_factory, cmd, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - try: - yield from protocol.waiter - except: - transport._kill_wait() - raise return Process(transport, protocol, loop) @coroutine @@ -241,9 +212,4 @@ program, *args, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - try: - yield from protocol.waiter - except: - transport._kill_wait() - raise return Process(transport, protocol, loop) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -16,6 +16,7 @@ from . import constants from . import coroutines from . import events +from . import futures from . import selector_events from . import selectors from . import transports @@ -175,16 +176,20 @@ stdin, stdout, stderr, bufsize, extra=None, **kwargs): with events.get_child_watcher() as watcher: + waiter = futures.Future(loop=self) transp = _UnixSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=extra, **kwargs) + waiter=waiter, extra=extra, + **kwargs) + + watcher.add_child_handler(transp.get_pid(), + self._child_watcher_callback, transp) try: - yield from transp._post_init() + yield from waiter except: transp.close() + yield from transp.wait() raise - watcher.add_child_handler(transp.get_pid(), - self._child_watcher_callback, transp) return transp @@ -774,7 +779,7 @@ pass def add_child_handler(self, pid, callback, *args): - self._callbacks[pid] = callback, args + self._callbacks[pid] = (callback, args) # Prevent a race condition in case the child is already terminated. self._do_waitpid(pid) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -366,13 +366,16 @@ def _make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra=None, **kwargs): + waiter = futures.Future(loop=self) transp = _WindowsSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=extra, **kwargs) + waiter=waiter, extra=extra, + **kwargs) try: - yield from transp._post_init() + yield from waiter except: transp.close() + yield from transp.wait() raise return transp diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1551,9 +1551,10 @@ stdin = transp.get_pipe_transport(0) stdin.write(b'Python The Winner') self.loop.run_until_complete(proto.got_data[1].wait()) - transp.close() + with test_utils.disable_logger(): + transp.close() self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) self.assertEqual(b'Python The Winner', proto.data[1]) def test_subprocess_interactive(self): @@ -1567,21 +1568,20 @@ self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - try: - stdin = transp.get_pipe_transport(0) - stdin.write(b'Python ') - self.loop.run_until_complete(proto.got_data[1].wait()) - proto.got_data[1].clear() - self.assertEqual(b'Python ', proto.data[1]) - - stdin.write(b'The Winner') - self.loop.run_until_complete(proto.got_data[1].wait()) - self.assertEqual(b'Python The Winner', proto.data[1]) - finally: + stdin = transp.get_pipe_transport(0) + stdin.write(b'Python ') + self.loop.run_until_complete(proto.got_data[1].wait()) + proto.got_data[1].clear() + self.assertEqual(b'Python ', proto.data[1]) + + stdin.write(b'The Winner') + self.loop.run_until_complete(proto.got_data[1].wait()) + self.assertEqual(b'Python The Winner', proto.data[1]) + + with test_utils.disable_logger(): transp.close() - self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) def test_subprocess_shell(self): connect = self.loop.subprocess_shell( @@ -1739,9 +1739,10 @@ # GetLastError()==ERROR_INVALID_NAME on Windows!?! (Using # WriteFile() we get ERROR_BROKEN_PIPE as expected.) self.assertEqual(b'ERR:OSError', proto.data[2]) - transp.close() + with test_utils.disable_logger(): + transp.close() self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) def test_subprocess_wait_no_same_group(self): # start the new process in a new session diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -4,6 +4,7 @@ from unittest import mock import asyncio +from asyncio import base_subprocess from asyncio import subprocess from asyncio import test_utils try: @@ -23,6 +24,70 @@ 'data = sys.stdin.buffer.read()', 'sys.stdout.buffer.write(data)'))] +class TestSubprocessTransport(base_subprocess.BaseSubprocessTransport): + def _start(self, *args, **kwargs): + self._proc = mock.Mock() + self._proc.stdin = None + self._proc.stdout = None + self._proc.stderr = None + + +class SubprocessTransportTests(test_utils.TestCase): + def setUp(self): + self.loop = self.new_test_loop() + self.set_event_loop(self.loop) + + + def create_transport(self, waiter=None): + protocol = mock.Mock() + protocol.connection_made._is_coroutine = False + protocol.process_exited._is_coroutine = False + transport = TestSubprocessTransport( + self.loop, protocol, ['test'], False, + None, None, None, 0, waiter=waiter) + return (transport, protocol) + + def test_close(self): + waiter = asyncio.Future(loop=self.loop) + transport, protocol = self.create_transport(waiter) + transport._process_exited(0) + transport.close() + + # The loop didn't run yet + self.assertFalse(protocol.connection_made.called) + + # methods must raise ProcessLookupError if the transport was closed + self.assertRaises(ValueError, transport.send_signal, signal.SIGTERM) + self.assertRaises(ValueError, transport.terminate) + self.assertRaises(ValueError, transport.kill) + + self.loop.run_until_complete(waiter) + + def test_proc_exited(self): + waiter = asyncio.Future(loop=self.loop) + transport, protocol = self.create_transport(waiter) + transport._process_exited(6) + self.loop.run_until_complete(waiter) + + self.assertEqual(transport.get_returncode(), 6) + + self.assertTrue(protocol.connection_made.called) + self.assertTrue(protocol.process_exited.called) + self.assertTrue(protocol.connection_lost.called) + self.assertEqual(protocol.connection_lost.call_args[0], (None,)) + + self.assertFalse(transport._closed) + self.assertIsNone(transport._loop) + self.assertIsNone(transport._proc) + self.assertIsNone(transport._protocol) + + # methods must raise ProcessLookupError if the process exited + self.assertRaises(ProcessLookupError, + transport.send_signal, signal.SIGTERM) + self.assertRaises(ProcessLookupError, transport.terminate) + self.assertRaises(ProcessLookupError, transport.kill) + + class SubprocessMixin: def test_stdin_stdout(self): -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:13:43 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:13:43 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150129231342.96068.6785@psf.io> https://hg.python.org/cpython/rev/c7f7270fbce7 changeset: 94382:c7f7270fbce7 parent: 94380:0701ffaa89f9 parent: 94381:89cde9b1a853 user: Victor Stinner date: Fri Jan 30 00:11:55 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/test/test_asyncio/test_subprocess.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -87,6 +87,8 @@ self.assertRaises(ProcessLookupError, transport.terminate) self.assertRaises(ProcessLookupError, transport.kill) + transport.close() + class SubprocessMixin: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:13:43 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:13:43 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?= =?utf-8?q?_ResourceWarning_in_test=5Fsubprocess=2Etest=5Fproc=5Fexit=28?= =?utf-8?q?=29?= Message-ID: <20150129231341.106370.2534@psf.io> https://hg.python.org/cpython/rev/89cde9b1a853 changeset: 94381:89cde9b1a853 branch: 3.4 parent: 94379:a5efd5021ca1 user: Victor Stinner date: Fri Jan 30 00:11:42 2015 +0100 summary: asyncio: Fix ResourceWarning in test_subprocess.test_proc_exit() files: Lib/test/test_asyncio/test_subprocess.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -87,6 +87,8 @@ self.assertRaises(ProcessLookupError, transport.terminate) self.assertRaises(ProcessLookupError, transport.kill) + transport.close() + class SubprocessMixin: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:17:48 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:17:48 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzQ3?= =?utf-8?q?=2C_asyncio=3A_Make_BaseSubprocessTransport=2Ewait=28=29_privat?= =?utf-8?q?e?= Message-ID: <20150129231740.34406.15393@psf.io> https://hg.python.org/cpython/rev/dadc372f46fa changeset: 94383:dadc372f46fa branch: 3.4 parent: 94381:89cde9b1a853 user: Victor Stinner date: Fri Jan 30 00:16:14 2015 +0100 summary: Issue #23347, asyncio: Make BaseSubprocessTransport.wait() private files: Lib/asyncio/base_subprocess.py | 2 +- Lib/asyncio/subprocess.py | 2 +- Lib/asyncio/unix_events.py | 2 +- Lib/asyncio/windows_events.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -208,7 +208,7 @@ waiter.set_result(returncode) self._exit_waiters = None - def wait(self): + def _wait(self): """Wait until the process exit and return the process return code. This method is a coroutine.""" diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -115,7 +115,7 @@ """Wait until the process exit and return the process return code. This method is a coroutine.""" - return (yield from self._transport.wait()) + return (yield from self._transport._wait()) def send_signal(self, signal): self._transport.send_signal(signal) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -188,7 +188,7 @@ yield from waiter except: transp.close() - yield from transp.wait() + yield from transp._wait() raise return transp diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -375,7 +375,7 @@ yield from waiter except: transp.close() - yield from transp.wait() + yield from transp._wait() raise return transp -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:17:48 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:17:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150129231740.96068.19622@psf.io> https://hg.python.org/cpython/rev/7e6340e67618 changeset: 94384:7e6340e67618 parent: 94382:c7f7270fbce7 parent: 94383:dadc372f46fa user: Victor Stinner date: Fri Jan 30 00:16:31 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_subprocess.py | 2 +- Lib/asyncio/subprocess.py | 2 +- Lib/asyncio/unix_events.py | 2 +- Lib/asyncio/windows_events.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -208,7 +208,7 @@ waiter.set_result(returncode) self._exit_waiters = None - def wait(self): + def _wait(self): """Wait until the process exit and return the process return code. This method is a coroutine.""" diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -115,7 +115,7 @@ """Wait until the process exit and return the process return code. This method is a coroutine.""" - return (yield from self._transport.wait()) + return (yield from self._transport._wait()) def send_signal(self, signal): self._transport.send_signal(signal) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -188,7 +188,7 @@ yield from waiter except: transp.close() - yield from transp.wait() + yield from transp._wait() raise return transp diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -375,7 +375,7 @@ yield from waiter except: transp.close() - yield from transp.wait() + yield from transp._wait() raise return transp -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:37:31 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:37:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?= Message-ID: <20150129233730.96074.90400@psf.io> https://hg.python.org/cpython/rev/23342733dd7e changeset: 94386:23342733dd7e parent: 94384:7e6340e67618 parent: 94385:21010940f8c1 user: Victor Stinner date: Fri Jan 30 00:37:22 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-sync.rst | 23 +++++++++++++++++++++++ 1 files changed, 23 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -4,6 +4,29 @@ Synchronization primitives ========================== +Locks: + +* :class:`Lock` +* :class:`Event` +* :class:`Condition` +* :class:`Semaphore` +* :class:`BoundedSemaphore` + +Queues: + +* :class:`Queue` +* :class:`PriorityQueue` +* :class:`LifoQueue` +* :class:`JoinableQueue` + +asyncio locks and queues API were designed to be close to classes of the +:mod:`threading` module (:class:`~threading.Lock`, :class:`~threading.Event`, +:class:`~threading.Condition`, :class:`~threading.Semaphore`, +:class:`~threading.BoundedSemaphore`) and the :mod:`queue` module +(:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, +:class:`~queue.LifoQueue`), but they have no *timeout* parameter. The +:func:`asyncio.wait_for` function can be used to cancel a task after a timeout. + Locks ----- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:37:31 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:37:31 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxOTYy?= =?utf-8?q?=2C_asyncio_doc=3A_Suggest_the_usage_of_wait=5Ffor=28=29_to_rep?= =?utf-8?q?lace?= Message-ID: <20150129233730.106379.96585@psf.io> https://hg.python.org/cpython/rev/21010940f8c1 changeset: 94385:21010940f8c1 branch: 3.4 parent: 94383:dadc372f46fa user: Victor Stinner date: Fri Jan 30 00:37:04 2015 +0100 summary: Issue #21962, asyncio doc: Suggest the usage of wait_for() to replace the lack of timeout parameter for locks and queues. files: Doc/library/asyncio-sync.rst | 23 +++++++++++++++++++++++ 1 files changed, 23 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -4,6 +4,29 @@ Synchronization primitives ========================== +Locks: + +* :class:`Lock` +* :class:`Event` +* :class:`Condition` +* :class:`Semaphore` +* :class:`BoundedSemaphore` + +Queues: + +* :class:`Queue` +* :class:`PriorityQueue` +* :class:`LifoQueue` +* :class:`JoinableQueue` + +asyncio locks and queues API were designed to be close to classes of the +:mod:`threading` module (:class:`~threading.Lock`, :class:`~threading.Event`, +:class:`~threading.Condition`, :class:`~threading.Semaphore`, +:class:`~threading.BoundedSemaphore`) and the :mod:`queue` module +(:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, +:class:`~queue.LifoQueue`), but they have no *timeout* parameter. The +:func:`asyncio.wait_for` function can be used to cancel a task after a timeout. + Locks ----- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:56:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:56:20 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbyBkb2M6?= =?utf-8?q?_add_a_section_about_task_cancellation?= Message-ID: <20150129235619.34398.42702@psf.io> https://hg.python.org/cpython/rev/173d4a8e1e74 changeset: 94387:173d4a8e1e74 branch: 3.4 parent: 94385:21010940f8c1 user: Victor Stinner date: Fri Jan 30 00:55:58 2015 +0100 summary: asyncio doc: add a section about task cancellation files: Doc/library/asyncio-dev.rst | 37 +++++++++++++++++++++++++ 1 files changed, 37 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -40,6 +40,43 @@ `. +Cancellation +------------ + +Cancellation of tasks is not common in classic programming. In asynchronous +programming, not only it is something common, but you have to prepare your +code to handle it. + +Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel` +method. The :func:`wait_for` function cancels the waited task when the timeout +occurs. There are many other cases where a task can be cancelled indirectly. + +Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method +of :class:`Future` if the future is cancelled: it would fail with an exception. +For example, write:: + + if not fut.cancelled(): + fut.set_result('done') + +Don't schedule directly a call to the :meth:`~Future.set_result` or the +:meth:`~Future.set_exception` method of a future with +:meth:`BaseEventLoop.call_soon`: the future can be cancelled before its method +is called. + +If you wait for a future, you should check early if the future was cancelled to +avoid useless operations. Example:: + + @coroutine + def slow_operation(fut): + if fut.cancelled(): + return + # ... slow computation ... + yield from fut + # ... + +The :func:`shield` function can also be used to ignore cancellation. + + .. _asyncio-multithreading: Concurrency and multithreading -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 00:56:20 2015 From: python-checkins at python.org (victor.stinner) Date: Thu, 29 Jan 2015 23:56:20 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?= Message-ID: <20150129235619.96074.83668@psf.io> https://hg.python.org/cpython/rev/faf94fc9d94c changeset: 94388:faf94fc9d94c parent: 94386:23342733dd7e parent: 94387:173d4a8e1e74 user: Victor Stinner date: Fri Jan 30 00:56:10 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-dev.rst | 37 +++++++++++++++++++++++++ 1 files changed, 37 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -40,6 +40,43 @@ `. +Cancellation +------------ + +Cancellation of tasks is not common in classic programming. In asynchronous +programming, not only it is something common, but you have to prepare your +code to handle it. + +Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel` +method. The :func:`wait_for` function cancels the waited task when the timeout +occurs. There are many other cases where a task can be cancelled indirectly. + +Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method +of :class:`Future` if the future is cancelled: it would fail with an exception. +For example, write:: + + if not fut.cancelled(): + fut.set_result('done') + +Don't schedule directly a call to the :meth:`~Future.set_result` or the +:meth:`~Future.set_exception` method of a future with +:meth:`BaseEventLoop.call_soon`: the future can be cancelled before its method +is called. + +If you wait for a future, you should check early if the future was cancelled to +avoid useless operations. Example:: + + @coroutine + def slow_operation(fut): + if fut.cancelled(): + return + # ... slow computation ... + yield from fut + # ... + +The :func:`shield` function can also be used to ignore cancellation. + + .. _asyncio-multithreading: Concurrency and multithreading -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 01:22:10 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 30 Jan 2015 00:22:10 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzQ3?= =?utf-8?b?LCBhc3luY2lvOiBzZW5kX3NpZ25hbCgpLCB0ZXJtaW5hdGUoKSwga2lsbCgp?= =?utf-8?q?_don=27t_check_if_the?= Message-ID: <20150130002202.34398.96542@psf.io> https://hg.python.org/cpython/rev/c1d92f7221a5 changeset: 94389:c1d92f7221a5 branch: 3.4 parent: 94387:173d4a8e1e74 user: Victor Stinner date: Fri Jan 30 01:20:44 2015 +0100 summary: Issue #23347, asyncio: send_signal(), terminate(), kill() don't check if the transport was closed. The check broken a Tulip example and this limitation is arbitrary. Check if _proc is None should be enough. Enhance also close(): do nothing when called the second time. files: Lib/asyncio/base_subprocess.py | 7 ++--- Lib/test/test_asyncio/test_subprocess.py | 16 ------------ 2 files changed, 3 insertions(+), 20 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -84,6 +84,8 @@ raise NotImplementedError def close(self): + if self._closed: + return self._closed = True for proto in self._pipes.values(): @@ -100,8 +102,7 @@ except ProcessLookupError: pass - # Don't clear the _proc reference yet because _post_init() may - # still run + # Don't clear the _proc reference yet: _post_init() may still run # On Python 3.3 and older, objects with a destructor part of a reference # cycle are never destroyed. It's not more the case on Python 3.4 thanks @@ -125,8 +126,6 @@ return None def _check_proc(self): - if self._closed: - raise ValueError("operation on closed transport") if self._proc is None: raise ProcessLookupError() diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -47,22 +47,6 @@ None, None, None, 0, waiter=waiter) return (transport, protocol) - def test_close(self): - waiter = asyncio.Future(loop=self.loop) - transport, protocol = self.create_transport(waiter) - transport._process_exited(0) - transport.close() - - # The loop didn't run yet - self.assertFalse(protocol.connection_made.called) - - # methods must raise ProcessLookupError if the transport was closed - self.assertRaises(ValueError, transport.send_signal, signal.SIGTERM) - self.assertRaises(ValueError, transport.terminate) - self.assertRaises(ValueError, transport.kill) - - self.loop.run_until_complete(waiter) - def test_proc_exited(self): waiter = asyncio.Future(loop=self.loop) transport, protocol = self.create_transport(waiter) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 01:22:09 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 30 Jan 2015 00:22:09 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20150130002203.96080.82712@psf.io> https://hg.python.org/cpython/rev/25c1d4aea1c1 changeset: 94390:25c1d4aea1c1 parent: 94388:faf94fc9d94c parent: 94389:c1d92f7221a5 user: Victor Stinner date: Fri Jan 30 01:21:06 2015 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_subprocess.py | 7 ++--- Lib/test/test_asyncio/test_subprocess.py | 16 ------------ 2 files changed, 3 insertions(+), 20 deletions(-) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -84,6 +84,8 @@ raise NotImplementedError def close(self): + if self._closed: + return self._closed = True for proto in self._pipes.values(): @@ -100,8 +102,7 @@ except ProcessLookupError: pass - # Don't clear the _proc reference yet because _post_init() may - # still run + # Don't clear the _proc reference yet: _post_init() may still run # On Python 3.3 and older, objects with a destructor part of a reference # cycle are never destroyed. It's not more the case on Python 3.4 thanks @@ -125,8 +126,6 @@ return None def _check_proc(self): - if self._closed: - raise ValueError("operation on closed transport") if self._proc is None: raise ProcessLookupError() diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -47,22 +47,6 @@ None, None, None, 0, waiter=waiter) return (transport, protocol) - def test_close(self): - waiter = asyncio.Future(loop=self.loop) - transport, protocol = self.create_transport(waiter) - transport._process_exited(0) - transport.close() - - # The loop didn't run yet - self.assertFalse(protocol.connection_made.called) - - # methods must raise ProcessLookupError if the transport was closed - self.assertRaises(ValueError, transport.send_signal, signal.SIGTERM) - self.assertRaises(ValueError, transport.terminate) - self.assertRaises(ValueError, transport.kill) - - self.loop.run_until_complete(waiter) - def test_proc_exited(self): waiter = asyncio.Future(loop=self.loop) transport, protocol = self.create_transport(waiter) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 01:39:10 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 30 Jan 2015 00:39:10 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbyBkb2M6?= =?utf-8?q?_document_the_new_ResourceWarning_warnings?= Message-ID: <20150130003909.39298.44034@psf.io> https://hg.python.org/cpython/rev/3c3f08e68c20 changeset: 94391:3c3f08e68c20 branch: 3.4 parent: 94389:c1d92f7221a5 user: Victor Stinner date: Fri Jan 30 01:35:14 2015 +0100 summary: asyncio doc: document the new ResourceWarning warnings files: Doc/library/asyncio-dev.rst | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -372,3 +372,14 @@ :ref:`Detect coroutine objects never scheduled `. + +Close transports +---------------- + +When a transport is no more needed, call its ``close()`` method to release +resources. + +If a transport (or an event loop) is not closed explicitly, a +:exc:`ResourceWarning` warning will be emitted in its destructor. The +:exc:`ResourceWarning` warnings are hidden by default: use the ``-Wd`` command +line option of Python to show them. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 01:39:10 2015 From: python-checkins at python.org (victor.stinner) Date: Fri, 30 Jan 2015 00:39:10 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?= Message-ID: <20150130003909.39294.63728@psf.io> https://hg.python.org/cpython/rev/582dd2ef9673 changeset: 94392:582dd2ef9673 parent: 94390:25c1d4aea1c1 parent: 94391:3c3f08e68c20 user: Victor Stinner date: Fri Jan 30 01:39:01 2015 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-dev.rst | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -372,3 +372,14 @@ :ref:`Detect coroutine objects never scheduled `. + +Close transports +---------------- + +When a transport is no more needed, call its ``close()`` method to release +resources. + +If a transport (or an event loop) is not closed explicitly, a +:exc:`ResourceWarning` warning will be emitted in its destructor. The +:exc:`ResourceWarning` warnings are hidden by default: use the ``-Wd`` command +line option of Python to show them. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 01:54:23 2015 From: python-checkins at python.org (r.david.murray) Date: Fri, 30 Jan 2015 00:54:23 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_asyncio_do?= =?utf-8?q?c_typo=2E?= Message-ID: <20150130005422.106309.15706@psf.io> https://hg.python.org/cpython/rev/fadfcb2d5a47 changeset: 94393:fadfcb2d5a47 branch: 3.4 parent: 94391:3c3f08e68c20 user: R David Murray date: Thu Jan 29 19:53:33 2015 -0500 summary: Fix asyncio doc typo. files: Doc/library/asyncio-eventloop.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -641,7 +641,7 @@ Server listening on sockets. Object created by the :meth:`BaseEventLoop.create_server` method and the - :func:`start_server` function. Don't instanciate the class directly. + :func:`start_server` function. Don't instantiate the class directly. .. method:: close() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 01:54:23 2015 From: python-checkins at python.org (r.david.murray) Date: Fri, 30 Jan 2015 00:54:23 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_Fix_asyncio_doc_typo=2E?= Message-ID: <20150130005422.25843.80610@psf.io> https://hg.python.org/cpython/rev/efb25925bd13 changeset: 94394:efb25925bd13 parent: 94392:582dd2ef9673 parent: 94393:fadfcb2d5a47 user: R David Murray date: Thu Jan 29 19:54:03 2015 -0500 summary: Merge: Fix asyncio doc typo. files: Doc/library/asyncio-eventloop.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -641,7 +641,7 @@ Server listening on sockets. Object created by the :meth:`BaseEventLoop.create_server` method and the - :func:`start_server` function. Don't instanciate the class directly. + :func:`start_server` function. Don't instantiate the class directly. .. method:: close() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 02:17:37 2015 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 30 Jan 2015 01:17:37 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Some_rephrasings=2E_No_techni?= =?utf-8?q?cal_change=2E?= Message-ID: <20150130011735.96072.10238@psf.io> https://hg.python.org/peps/rev/0eee0d4f2b1d changeset: 5681:0eee0d4f2b1d user: Antoine Pitrou date: Fri Jan 30 02:17:26 2015 +0100 summary: Some rephrasings. No technical change. files: pep-0475.txt | 186 +++++++++++++++++++++----------------- 1 files changed, 101 insertions(+), 85 deletions(-) diff --git a/pep-0475.txt b/pep-0475.txt --- a/pep-0475.txt +++ b/pep-0475.txt @@ -14,8 +14,12 @@ Abstract ======== -Retry system calls failing with the ``EINTR`` error and recompute -timeout if needed. +System call wrappers provided in the standard library should be retried +automatically when they fail with ``EINTR``, to relieve application code +from the burden of doing so. + +By system calls, we mean the functions exposed by the standard C library +pertaining to I/O or handling of other system resources. Rationale @@ -24,10 +28,10 @@ Interrupted system calls ------------------------ -On POSIX systems, signals are common. Your program must be prepared to -handle them. Examples of signals: +On POSIX systems, signals are common. Code calling system calls must be +prepared to handle them. Examples of signals: -* The most common signal is ``SIGINT``, signal sent when CTRL+c is +* The most common signal is ``SIGINT``, the signal sent when CTRL+c is pressed. By default, Python raises a ``KeyboardInterrupt`` exception when this signal is received. * When running subprocesses, the ``SIGCHLD`` signal is sent when a @@ -37,37 +41,35 @@ * Putting the application in background (ex: press CTRL-z and then type the ``bg`` command) sends the ``SIGCONT`` signal. -Writing a signal handler is difficult, only "async-signal safe" -functions can be called. For example, ``printf()`` and ``malloc()`` -are not async-signal safe. When a signal is sent to a process calling -a system call, the system call can fail with the ``EINTR`` error to +Writing a C signal handler is difficult: only "async-signal-safe" +functions can be called (for example, ``printf()`` and ``malloc()`` +are not async-signal safe), and there are issues with reentrancy. +Therefore, when a signal is received by a process during the execution +of a system call, the system call can fail with the ``EINTR`` error to give the program an opportunity to handle the signal without the -restriction on signal safe functions. Depending on the platform, on -the system call and the ``SA_RESTART`` flag, the system call may or -may not fail with ``EINTR``. +restriction on signal-safe functions. -If the signal handler was set with the ``SA_RESTART`` flag set, the -kernel retries some the system call instead of failing with -``EINTR``. For example, ``read()`` is retried, whereas ``select()`` is -not retried. The Python function ``signal.signal()`` clears the -``SA_RESTART`` flag when setting the signal handler: all system calls -should fail with ``EINTR`` in Python. +This behaviour is system-dependent: on certain systems, using the +``SA_RESTART`` flag, some system calls are retried automatically instead +of failing with ``EINTR``. Regardless, Python's ``signal.signal()`` +function clears the ``SA_RESTART`` flag when setting the signal handler: +all system calls will probably fail with ``EINTR`` in Python. -The problem is that handling ``EINTR`` should be done for all system -calls. The problem is similar to handling errors in the C language -which does not have exceptions: you must check all function returns to -check for error, and usually duplicate the code checking for errors. -Python does not have this issue, it uses exceptions to notify errors. +Since receiving a signal is a non-exceptional occurrence, robust POSIX code +must be prepared to handle ``EINTR`` (which, in most cases, means +retry in a loop in the hope that the call eventually succeeds). +Without special support from Python, this can make application code +much more verbose than it needs to be. Status in Python 3.4 -------------------- -In Python 3.4, the code to handle the ``InterruptedError`` -exception (``EINTR`` error) is duplicated on case by case. Only a few -Python modules handle this exception, and fixes usually took several -years to cover a whole module. Example of code retrying -``file.read()`` on ``InterruptedError``:: +In Python 3.4, handling the ``InterruptedError`` exception (``EINTR``'s +dedicated exception class) is duplicated at every call site on a case by +case basis. Only a few Python modules actually handle this exception, +and fixes usually took several years to cover a whole module. Example of +code retrying ``file.read()`` on ``InterruptedError``:: while True: try: @@ -76,7 +78,7 @@ except InterruptedError: continue -List of Python modules of the standard library which handle +List of Python modules in the standard library which handle ``InterruptedError``: * ``asyncio`` @@ -88,16 +90,17 @@ * ``socketserver`` * ``subprocess`` -Other programming languages like Perl, Java and Go already retry -system calls failing with ``EINTR``. +Other programming languages like Perl, Java and Go retry system calls +failing with ``EINTR`` at a lower level, so that libraries and applications +needn't bother. Use Case 1: Don't Bother With Signals ------------------------------------- In most cases, you don't want to be interrupted by signals and you -don't expect to get ``InterruptedError`` exceptions. For example, do -you really want to write such complex code for an "Hello World" +don't expect to get ``InterruptedError`` exceptions. For example, do +you really want to write such complex code for a "Hello World" example? :: @@ -110,48 +113,59 @@ continue ``InterruptedError`` can happen in unexpected places. For example, -``os.close()`` and ``FileIO.close()`` can raises ``InterruptedError``: +``os.close()`` and ``FileIO.close()`` may raise ``InterruptedError``: see the article `close() and EINTR `_. The `Python issues related to EINTR`_ section below gives examples of -bugs caused by "EINTR". +bugs caused by ``EINTR``. -The expectation is that Python hides the ``InterruptedError``: retry -system calls failing with the ``EINTR`` error. +The expectation in this use case is that Python hides the +``InterruptedError`` and retries system calls automatically. Use Case 2: Be notified of signals as soon as possible ------------------------------------------------------ -Sometimes, you expect some signals and you want to handle them as soon -as possible. For example, you may want to quit immediatly a program -using the ``CTRL+c`` keyboard shortcut. +Sometimes yet, you expect some signals and you want to handle them as +soon as possible. For example, you may want to immediately quit a +program using the ``CTRL+c`` keyboard shortcut. -Some signals are not interesting and should not interrupt the the -application. There are two options to only interrupt an application -on some signals: +Besides, some signals are not interesting and should not disrupt the +application. There are two options to interrupt an application on +only *some* signals: -* Raise an exception in the signal handler, like ``KeyboardInterrupt`` for - ``SIGINT`` -* Use a I/O multiplexing function like ``select()`` with the Python - signal "wakeup" file descriptor: see the function - ``signal.set_wakeup_fd()``. +* Set up a custom signal signal handler which raises an exception, such as + ``KeyboardInterrupt`` for ``SIGINT``. +* Use a I/O multiplexing function like ``select()`` together with Python's + signal wakeup file descriptor: see the function ``signal.set_wakeup_fd()``. +The expectation in this use case is for the Python signal handler to be +executed timely, and the system call to fail if the handler raised an +exception -- otherwise restart. -Proposition -=========== -If a system call fails with ``EINTR``, Python must call signal -handlers: call ``PyErr_CheckSignals()``. If a signal handler raises -an exception, the Python function fails with the exception. -Otherwise, the system call is retried. If the system call takes a -timeout parameter, the timeout is recomputed. +Proposal +======== + +This PEP proposes to handle EINTR and retries at the lowest level, i.e. +in the wrappers provided by the stdlib (as opposed to higher-level +libraries and applications). + +Specifically, when a system call fails with ``EINTR``, its Python wrapper +must call the given signal handler (using ``PyErr_CheckSignals()``). +If the signal handler raises an exception, the Python wrapper bails out +and fails with the exception. + +If the signal handler returns successfully, the Python wrapper retries the +system call automatically. If the system call involves a timeout parameter, +the timeout is recomputed. Modified functions ------------------ -Example of functions that need to be modified: +Example of standard library functions that need to be modified to comply +with this PEP: * ``os.read()``, ``io.FileIO.read()``, ``io.FileIO.readinto()`` * ``os.write()``, ``io.FileIO.write()`` @@ -170,38 +184,40 @@ * ``select.kqueue.control()`` * ``selectors.SelectSelector.select()`` and other selector classes -Note: The ``selector`` module already retries on ``InterruptedError``, but it -doesn't recompute the timeout yet. +(note: the ``selector`` module already retries on ``InterruptedError``, but it +doesn't recompute the timeout yet) - -InterruptedError ----------------- +InterruptedError handling +------------------------- Since interrupted system calls are automatically retried, the -``InterruptedError`` exception should not occur anymore. The code handling -``InterruptedError`` can be removed from in the standard library to simply the -code. +``InterruptedError`` exception should not occur anymore when calling those +system calls. Therefore, manual handling of ``InterruptedError`` as +described in `Status in Python 3.4`_ can be removed, which will simplify +standard library code. -Backward Compatibility +Backward compatibility ====================== Applications relying on the fact that system calls are interrupted -with ``InterruptedError`` will hang. The authors of this PEP don't -think that such application exist. +with ``InterruptedError`` will hang. The authors of this PEP don't +think that such applications exist, since they would be exposed to +other issues such as race conditions (there is an opportunity for deadlock +if the signal comes before the system call). Besides, such code would +be non-portable. -If such applications exist, they are not portable and are subject to -race conditions (deadlock if the signal comes before the system call). -These applications must be fixed to handle signals differently, to -have a reliable behaviour on all platforms and all Python versions. -For example, use a signal handler which raises an exception, or use a -wakeup file descriptor. +In any case, those applications must be fixed to handle signals differently, +to have a reliable behaviour on all platforms and all Python versions. +A possible strategy is to set up a signal handler raising a well-defined +exception, or use a wakeup file descriptor. For applications using event loops, ``signal.set_wakeup_fd()`` is the -recommanded option to handle signals. The signal handler writes signal -numbers into the file descriptor and the event loop is awaken to read -them. The event loop can handle these signals without the restriction -of signal handlers. +recommanded option to handle signals. Python's low-level signal handler +will write signal numbers into the file descriptor and the event loop +will be awaken to read them. The event loop can handle those signals +without the restriction of signal handlers (for example, the loop can +be woken up in any thread, not just the main thread). Appendix @@ -212,12 +228,12 @@ Since Python 3.3, ``signal.set_wakeup_fd()`` writes the signal number into the file descriptor, whereas it only wrote a null byte before. -It becomes possible to handle different signals using the wakeup file +It becomes possible to distinguish between signals using the wakeup file descriptor. -Linux has a ``signalfd()`` which provides more information on each -signal. For example, it's possible to know the pid and uid who sent -the signal. This function is not exposed in Python yet (see the +Linux has a ``signalfd()`` system call which provides more information on +each signal. For example, it's possible to know the pid and uid who sent +the signal. This function is not exposed in Python yet (see `issue 12304 `_). On Unix, the ``asyncio`` module uses the wakeup file descriptor to @@ -227,11 +243,11 @@ Multithreading -------------- -A C signal handler can be called from any thread, but the Python -signal handler should only be called in the main thread. +A C signal handler can be called from any thread, but Python +signal handlers will always be called in the main Python thread. -Python has a ``PyErr_SetInterrupt()`` function which calls the -``SIGINT`` signal handler to interrupt the Python main thread. +Python's C API provides the ``PyErr_SetInterrupt()`` function which calls +the ``SIGINT`` signal handler in order to interrupt the main Python thread. Signals on Windows -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 30 07:02:34 2015 From: python-checkins at python.org (raymond.hettinger) Date: Fri, 30 Jan 2015 06:02:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Move_the_set_search_finger?= =?utf-8?q?_before_the_smalltable=2E?= Message-ID: <20150130060223.34402.59720@psf.io> https://hg.python.org/cpython/rev/b5c95ff0f9a4 changeset: 94395:b5c95ff0f9a4 parent: 94365:fee78636a3b1 user: Raymond Hettinger date: Thu Jan 29 22:00:32 2015 -0800 summary: Move the set search finger before the smalltable. files: Include/setobject.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Include/setobject.h b/Include/setobject.h --- a/Include/setobject.h +++ b/Include/setobject.h @@ -57,9 +57,9 @@ */ setentry *table; Py_hash_t hash; /* Only used by frozenset objects */ + Py_ssize_t finger; /* Search finger for pop() */ + setentry smalltable[PySet_MINSIZE]; - - Py_ssize_t finger; /* Search finger for pop() */ PyObject *weakreflist; /* List of weak references */ } PySetObject; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 07:02:34 2015 From: python-checkins at python.org (raymond.hettinger) Date: Fri, 30 Jan 2015 06:02:34 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <20150130060223.25851.1749@psf.io> https://hg.python.org/cpython/rev/424314dd2381 changeset: 94396:424314dd2381 parent: 94395:b5c95ff0f9a4 parent: 94394:efb25925bd13 user: Raymond Hettinger date: Thu Jan 29 22:02:17 2015 -0800 summary: merge files: Doc/library/asyncio-dev.rst | 48 +++ Doc/library/asyncio-eventloop.rst | 2 +- Doc/library/asyncio-protocol.rst | 8 + Doc/library/asyncio-sync.rst | 23 + Doc/whatsnew/3.5.rst | 6 + Include/memoryobject.h | 4 - Lib/asyncio/base_events.py | 11 + Lib/asyncio/base_subprocess.py | 124 ++++++--- Lib/asyncio/futures.py | 6 +- Lib/asyncio/proactor_events.py | 11 + Lib/asyncio/protocols.py | 5 + Lib/asyncio/selector_events.py | 73 +++++- Lib/asyncio/sslproto.py | 13 + Lib/asyncio/subprocess.py | 40 +-- Lib/asyncio/tasks.py | 2 +- Lib/asyncio/unix_events.py | 51 +++- Lib/asyncio/windows_events.py | 7 +- Lib/asyncio/windows_utils.py | 6 +- Lib/test/test_asyncio/test_events.py | 72 +++-- Lib/test/test_asyncio/test_proactor_events.py | 6 +- Lib/test/test_asyncio/test_selector_events.py | 15 +- Lib/test/test_asyncio/test_sslproto.py | 7 +- Lib/test/test_asyncio/test_subprocess.py | 51 ++++ Lib/test/test_asyncio/test_unix_events.py | 31 +- Lib/test/test_memoryview.py | 21 + Lib/test/test_sys.py | 2 +- Objects/memoryobject.c | 56 ++++- 27 files changed, 529 insertions(+), 172 deletions(-) diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -40,6 +40,43 @@ `. +Cancellation +------------ + +Cancellation of tasks is not common in classic programming. In asynchronous +programming, not only it is something common, but you have to prepare your +code to handle it. + +Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel` +method. The :func:`wait_for` function cancels the waited task when the timeout +occurs. There are many other cases where a task can be cancelled indirectly. + +Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method +of :class:`Future` if the future is cancelled: it would fail with an exception. +For example, write:: + + if not fut.cancelled(): + fut.set_result('done') + +Don't schedule directly a call to the :meth:`~Future.set_result` or the +:meth:`~Future.set_exception` method of a future with +:meth:`BaseEventLoop.call_soon`: the future can be cancelled before its method +is called. + +If you wait for a future, you should check early if the future was cancelled to +avoid useless operations. Example:: + + @coroutine + def slow_operation(fut): + if fut.cancelled(): + return + # ... slow computation ... + yield from fut + # ... + +The :func:`shield` function can also be used to ignore cancellation. + + .. _asyncio-multithreading: Concurrency and multithreading @@ -335,3 +372,14 @@ :ref:`Detect coroutine objects never scheduled `. + +Close transports +---------------- + +When a transport is no more needed, call its ``close()`` method to release +resources. + +If a transport (or an event loop) is not closed explicitly, a +:exc:`ResourceWarning` warning will be emitted in its destructor. The +:exc:`ResourceWarning` warnings are hidden by default: use the ``-Wd`` command +line option of Python to show them. diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -641,7 +641,7 @@ Server listening on sockets. Object created by the :meth:`BaseEventLoop.create_server` method and the - :func:`start_server` function. Don't instanciate the class directly. + :func:`start_server` function. Don't instantiate the class directly. .. method:: close() diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -374,6 +374,14 @@ a connection. However, :meth:`eof_received` is called at most once and, if called, :meth:`data_received` won't be called after it. +State machine: + + start -> :meth:`~BaseProtocol.connection_made` + [-> :meth:`~Protocol.data_received` \*] + [-> :meth:`~Protocol.eof_received` ?] + -> :meth:`~BaseProtocol.connection_lost` -> end + + Datagram protocols ------------------ diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -4,6 +4,29 @@ Synchronization primitives ========================== +Locks: + +* :class:`Lock` +* :class:`Event` +* :class:`Condition` +* :class:`Semaphore` +* :class:`BoundedSemaphore` + +Queues: + +* :class:`Queue` +* :class:`PriorityQueue` +* :class:`LifoQueue` +* :class:`JoinableQueue` + +asyncio locks and queues API were designed to be close to classes of the +:mod:`threading` module (:class:`~threading.Lock`, :class:`~threading.Event`, +:class:`~threading.Condition`, :class:`~threading.Semaphore`, +:class:`~threading.BoundedSemaphore`) and the :mod:`queue` module +(:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, +:class:`~queue.LifoQueue`), but they have no *timeout* parameter. The +:func:`asyncio.wait_for` function can be used to cancel a task after a timeout. + Locks ----- diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -485,6 +485,12 @@ Changes in the C API -------------------- +* The undocumented :c:member:`~PyMemoryViewObject.format` member of the + (non-public) :c:type:`PyMemoryViewObject` structure has been removed. + + All extensions relying on the relevant parts in ``memoryobject.h`` + must be rebuilt. + * The :c:type:`PyMemAllocator` structure was renamed to :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. diff --git a/Include/memoryobject.h b/Include/memoryobject.h --- a/Include/memoryobject.h +++ b/Include/memoryobject.h @@ -45,9 +45,6 @@ } _PyManagedBufferObject; -/* static storage used for casting between formats */ -#define _Py_MEMORYVIEW_MAX_FORMAT 3 /* must be >= 3 */ - /* memoryview state flags */ #define _Py_MEMORYVIEW_RELEASED 0x001 /* access to master buffer blocked */ #define _Py_MEMORYVIEW_C 0x002 /* C-contiguous layout */ @@ -62,7 +59,6 @@ int flags; /* state flags */ Py_ssize_t exports; /* number of buffer re-exports */ Py_buffer view; /* private copy of the exporter's view */ - char format[_Py_MEMORYVIEW_MAX_FORMAT]; /* used for casting */ PyObject *weakreflist; Py_ssize_t ob_array[1]; /* shape, strides, suboffsets */ } PyMemoryViewObject; diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -26,6 +26,7 @@ import time import traceback import sys +import warnings from . import coroutines from . import events @@ -333,6 +334,16 @@ """Returns True if the event loop was closed.""" return self._closed + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self.is_closed(): + warnings.warn("unclosed event loop %r" % self, ResourceWarning) + if not self.is_running(): + self.close() + def is_running(self): """Returns True if the event loop is running.""" return (self._owner is not None) diff --git a/Lib/asyncio/base_subprocess.py b/Lib/asyncio/base_subprocess.py --- a/Lib/asyncio/base_subprocess.py +++ b/Lib/asyncio/base_subprocess.py @@ -1,6 +1,9 @@ import collections import subprocess +import sys +import warnings +from . import futures from . import protocols from . import transports from .coroutines import coroutine @@ -11,26 +14,32 @@ def __init__(self, loop, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=None, **kwargs): + waiter=None, extra=None, **kwargs): super().__init__(extra) + self._closed = False self._protocol = protocol self._loop = loop + self._proc = None self._pid = None + self._returncode = None + self._exit_waiters = [] + self._pending_calls = collections.deque() + self._pipes = {} + self._finished = False - self._pipes = {} if stdin == subprocess.PIPE: self._pipes[0] = None if stdout == subprocess.PIPE: self._pipes[1] = None if stderr == subprocess.PIPE: self._pipes[2] = None - self._pending_calls = collections.deque() - self._finished = False - self._returncode = None + + # Create the child process: set the _proc attribute self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, stderr=stderr, bufsize=bufsize, **kwargs) self._pid = self._proc.pid self._extra['subprocess'] = self._proc + if self._loop.get_debug(): if isinstance(args, (bytes, str)): program = args @@ -39,8 +48,13 @@ logger.debug('process %r created: pid %s', program, self._pid) + self._loop.create_task(self._connect_pipes(waiter)) + def __repr__(self): - info = [self.__class__.__name__, 'pid=%s' % self._pid] + info = [self.__class__.__name__] + if self._closed: + info.append('closed') + info.append('pid=%s' % self._pid) if self._returncode is not None: info.append('returncode=%s' % self._returncode) @@ -70,12 +84,34 @@ raise NotImplementedError def close(self): + if self._closed: + return + self._closed = True + for proto in self._pipes.values(): if proto is None: continue proto.pipe.close() - if self._returncode is None: - self.terminate() + + if self._proc is not None and self._returncode is None: + if self._loop.get_debug(): + logger.warning('Close running child process: kill %r', self) + + try: + self._proc.kill() + except ProcessLookupError: + pass + + # Don't clear the _proc reference yet: _post_init() may still run + + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self._closed: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() def get_pid(self): return self._pid @@ -89,58 +125,40 @@ else: return None + def _check_proc(self): + if self._proc is None: + raise ProcessLookupError() + def send_signal(self, signal): + self._check_proc() self._proc.send_signal(signal) def terminate(self): + self._check_proc() self._proc.terminate() def kill(self): + self._check_proc() self._proc.kill() - def _kill_wait(self): - """Close pipes, kill the subprocess and read its return status. - - Function called when an exception is raised during the creation - of a subprocess. - """ - if self._loop.get_debug(): - logger.warning('Exception during subprocess creation, ' - 'kill the subprocess %r', - self, - exc_info=True) - - proc = self._proc - if proc.stdout: - proc.stdout.close() - if proc.stderr: - proc.stderr.close() - if proc.stdin: - proc.stdin.close() - - try: - proc.kill() - except ProcessLookupError: - pass - self._returncode = proc.wait() - - self.close() - @coroutine - def _post_init(self): + def _connect_pipes(self, waiter): try: proc = self._proc loop = self._loop + if proc.stdin is not None: _, pipe = yield from loop.connect_write_pipe( lambda: WriteSubprocessPipeProto(self, 0), proc.stdin) self._pipes[0] = pipe + if proc.stdout is not None: _, pipe = yield from loop.connect_read_pipe( lambda: ReadSubprocessPipeProto(self, 1), proc.stdout) self._pipes[1] = pipe + if proc.stderr is not None: _, pipe = yield from loop.connect_read_pipe( lambda: ReadSubprocessPipeProto(self, 2), @@ -149,13 +167,16 @@ assert self._pending_calls is not None - self._loop.call_soon(self._protocol.connection_made, self) + loop.call_soon(self._protocol.connection_made, self) for callback, data in self._pending_calls: - self._loop.call_soon(callback, *data) + loop.call_soon(callback, *data) self._pending_calls = None - except: - self._kill_wait() - raise + except Exception as exc: + if waiter is not None and not waiter.cancelled(): + waiter.set_exception(exc) + else: + if waiter is not None and not waiter.cancelled(): + waiter.set_result(None) def _call(self, cb, *data): if self._pending_calls is not None: @@ -180,6 +201,23 @@ self._call(self._protocol.process_exited) self._try_finish() + # wake up futures waiting for wait() + for waiter in self._exit_waiters: + if not waiter.cancelled(): + waiter.set_result(returncode) + self._exit_waiters = None + + def _wait(self): + """Wait until the process exit and return the process return code. + + This method is a coroutine.""" + if self._returncode is not None: + return self._returncode + + waiter = futures.Future(loop=self._loop) + self._exit_waiters.append(waiter) + return (yield from waiter) + def _try_finish(self): assert not self._finished if self._returncode is None: @@ -193,9 +231,9 @@ try: self._protocol.connection_lost(exc) finally: + self._loop = None self._proc = None self._protocol = None - self._loop = None class WriteSubprocessPipeProto(protocols.BaseProtocol): diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -195,9 +195,9 @@ info = self._repr_info() return '<%s %s>' % (self.__class__.__name__, ' '.join(info)) - # On Python 3.3 or older, objects with a destructor part of a reference - # cycle are never destroyed. It's not more the case on Python 3.4 thanks to - # the PEP 442. + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. if _PY34: def __del__(self): if not self._log_traceback: diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -7,6 +7,8 @@ __all__ = ['BaseProactorEventLoop'] import socket +import sys +import warnings from . import base_events from . import constants @@ -74,6 +76,15 @@ self._read_fut.cancel() self._read_fut = None + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._sock is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() + def _fatal_error(self, exc, message='Fatal error on pipe transport'): if isinstance(exc, (BrokenPipeError, ConnectionResetError)): if self._loop.get_debug(): diff --git a/Lib/asyncio/protocols.py b/Lib/asyncio/protocols.py --- a/Lib/asyncio/protocols.py +++ b/Lib/asyncio/protocols.py @@ -78,6 +78,11 @@ State machine of calls: start -> CM [-> DR*] [-> ER?] -> CL -> end + + * CM: connection_made() + * DR: data_received() + * ER: eof_received() + * CL: connection_lost() """ def data_received(self, data): diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -10,6 +10,8 @@ import errno import functools import socket +import sys +import warnings try: import ssl except ImportError: # pragma: no cover @@ -22,6 +24,7 @@ from . import selectors from . import transports from . import sslproto +from .coroutines import coroutine from .log import logger @@ -181,16 +184,47 @@ else: raise # The event loop will catch, log and ignore it. else: + extra = {'peername': addr} + accept = self._accept_connection2(protocol_factory, conn, extra, + sslcontext, server) + self.create_task(accept) + + @coroutine + def _accept_connection2(self, protocol_factory, conn, extra, + sslcontext=None, server=None): + protocol = None + transport = None + try: protocol = protocol_factory() + waiter = futures.Future(loop=self) if sslcontext: - self._make_ssl_transport( - conn, protocol, sslcontext, - server_side=True, extra={'peername': addr}, server=server) + transport = self._make_ssl_transport( + conn, protocol, sslcontext, waiter=waiter, + server_side=True, extra=extra, server=server) else: - self._make_socket_transport( - conn, protocol , extra={'peername': addr}, + transport = self._make_socket_transport( + conn, protocol, waiter=waiter, extra=extra, server=server) - # It's now up to the protocol to handle the connection. + + try: + yield from waiter + except: + transport.close() + raise + + # It's now up to the protocol to handle the connection. + except Exception as exc: + if self.get_debug(): + context = { + 'message': ('Error on transport creation ' + 'for incoming connection'), + 'exception': exc, + } + if protocol is not None: + context['protocol'] = protocol + if transport is not None: + context['transport'] = transport + self.call_exception_handler(context) def add_reader(self, fd, callback, *args): """Add a reader callback.""" @@ -467,7 +501,12 @@ _buffer_factory = bytearray # Constructs initial value for self._buffer. - def __init__(self, loop, sock, protocol, extra, server=None): + # Attribute used in the destructor: it must be set even if the constructor + # is not called (see _SelectorSslTransport which may start by raising an + # exception) + _sock = None + + def __init__(self, loop, sock, protocol, extra=None, server=None): super().__init__(extra, loop) self._extra['socket'] = sock self._extra['sockname'] = sock.getsockname() @@ -479,6 +518,7 @@ self._sock = sock self._sock_fd = sock.fileno() self._protocol = protocol + self._protocol_connected = True self._server = server self._buffer = self._buffer_factory() self._conn_lost = 0 # Set when call to connection_lost scheduled. @@ -526,6 +566,15 @@ self._conn_lost += 1 self._loop.call_soon(self._call_connection_lost, None) + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._sock is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._sock.close() + def _fatal_error(self, exc, message='Fatal error on transport'): # Should be called from exception handler only. if isinstance(exc, (BrokenPipeError, @@ -555,7 +604,8 @@ def _call_connection_lost(self, exc): try: - self._protocol.connection_lost(exc) + if self._protocol_connected: + self._protocol.connection_lost(exc) finally: self._sock.close() self._sock = None @@ -718,6 +768,8 @@ sslsock = sslcontext.wrap_socket(rawsock, **wrap_kwargs) super().__init__(loop, sslsock, protocol, extra, server) + # the protocol connection is only made after the SSL handshake + self._protocol_connected = False self._server_hostname = server_hostname self._waiter = waiter @@ -797,6 +849,7 @@ self._read_wants_write = False self._write_wants_read = False self._loop.add_reader(self._sock_fd, self._read_ready) + self._protocol_connected = True self._loop.call_soon(self._protocol.connection_made, self) # only wake up the waiter when connection_made() has been called self._loop.call_soon(self._wakeup_waiter) @@ -928,8 +981,10 @@ waiter=None, extra=None): super().__init__(loop, sock, protocol, extra) self._address = address - self._loop.add_reader(self._sock_fd, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._sock_fd, self._read_ready) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) diff --git a/Lib/asyncio/sslproto.py b/Lib/asyncio/sslproto.py --- a/Lib/asyncio/sslproto.py +++ b/Lib/asyncio/sslproto.py @@ -1,4 +1,6 @@ import collections +import sys +import warnings try: import ssl except ImportError: # pragma: no cover @@ -295,6 +297,7 @@ self._loop = loop self._ssl_protocol = ssl_protocol self._app_protocol = app_protocol + self._closed = False def get_extra_info(self, name, default=None): """Get optional transport information.""" @@ -308,8 +311,18 @@ protocol's connection_lost() method will (eventually) called with None as its argument. """ + self._closed = True self._ssl_protocol._start_shutdown() + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if not self._closed: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self.close() + def pause_reading(self): """Pause the receiving end. diff --git a/Lib/asyncio/subprocess.py b/Lib/asyncio/subprocess.py --- a/Lib/asyncio/subprocess.py +++ b/Lib/asyncio/subprocess.py @@ -25,8 +25,6 @@ super().__init__(loop=loop) self._limit = limit self.stdin = self.stdout = self.stderr = None - self.waiter = futures.Future(loop=loop) - self._waiters = collections.deque() self._transport = None def __repr__(self): @@ -61,9 +59,6 @@ reader=None, loop=self._loop) - if not self.waiter.cancelled(): - self.waiter.set_result(None) - def pipe_data_received(self, fd, data): if fd == 1: reader = self.stdout @@ -94,16 +89,9 @@ reader.set_exception(exc) def process_exited(self): - returncode = self._transport.get_returncode() self._transport.close() self._transport = None - # wake up futures waiting for wait() - while self._waiters: - waiter = self._waiters.popleft() - if not waiter.cancelled(): - waiter.set_result(returncode) - class Process: def __init__(self, transport, protocol, loop): @@ -124,30 +112,18 @@ @coroutine def wait(self): - """Wait until the process exit and return the process return code.""" - returncode = self._transport.get_returncode() - if returncode is not None: - return returncode + """Wait until the process exit and return the process return code. - waiter = futures.Future(loop=self._loop) - self._protocol._waiters.append(waiter) - yield from waiter - return waiter.result() - - def _check_alive(self): - if self._transport.get_returncode() is not None: - raise ProcessLookupError() + This method is a coroutine.""" + return (yield from self._transport._wait()) def send_signal(self, signal): - self._check_alive() self._transport.send_signal(signal) def terminate(self): - self._check_alive() self._transport.terminate() def kill(self): - self._check_alive() self._transport.kill() @coroutine @@ -221,11 +197,6 @@ protocol_factory, cmd, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - try: - yield from protocol.waiter - except: - transport._kill_wait() - raise return Process(transport, protocol, loop) @coroutine @@ -241,9 +212,4 @@ program, *args, stdin=stdin, stdout=stdout, stderr=stderr, **kwds) - try: - yield from protocol.waiter - except: - transport._kill_wait() - raise return Process(transport, protocol, loop) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -592,7 +592,7 @@ fut.exception() return - if fut._state == futures._CANCELLED: + if fut.cancelled(): res = futures.CancelledError() if not return_exceptions: outer.set_exception(res) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -8,6 +8,7 @@ import subprocess import sys import threading +import warnings from . import base_events @@ -15,6 +16,7 @@ from . import constants from . import coroutines from . import events +from . import futures from . import selector_events from . import selectors from . import transports @@ -174,16 +176,20 @@ stdin, stdout, stderr, bufsize, extra=None, **kwargs): with events.get_child_watcher() as watcher: + waiter = futures.Future(loop=self) transp = _UnixSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=extra, **kwargs) + waiter=waiter, extra=extra, + **kwargs) + + watcher.add_child_handler(transp.get_pid(), + self._child_watcher_callback, transp) try: - yield from transp._post_init() + yield from waiter except: transp.close() + yield from transp._wait() raise - watcher.add_child_handler(transp.get_pid(), - self._child_watcher_callback, transp) return transp @@ -298,8 +304,10 @@ _set_nonblocking(self._fileno) self._protocol = protocol self._closing = False - self._loop.add_reader(self._fileno, self._read_ready) self._loop.call_soon(self._protocol.connection_made, self) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._fileno, self._read_ready) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) @@ -351,6 +359,15 @@ if not self._closing: self._close(None) + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._pipe is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._pipe.close() + def _fatal_error(self, exc, message='Fatal error on pipe transport'): # should be called by exception handler only if (isinstance(exc, OSError) and exc.errno == errno.EIO): @@ -401,13 +418,16 @@ self._conn_lost = 0 self._closing = False # Set when close() or write_eof() called. - # On AIX, the reader trick only works for sockets. - # On other platforms it works for pipes and sockets. - # (Exception: OS X 10.4? Issue #19294.) + self._loop.call_soon(self._protocol.connection_made, self) + + # On AIX, the reader trick (to be notified when the read end of the + # socket is closed) only works for sockets. On other platforms it + # works for pipes and sockets. (Exception: OS X 10.4? Issue #19294.) if is_socket or not sys.platform.startswith("aix"): - self._loop.add_reader(self._fileno, self._read_ready) + # only start reading when connection_made() has been called + self._loop.call_soon(self._loop.add_reader, + self._fileno, self._read_ready) - self._loop.call_soon(self._protocol.connection_made, self) if waiter is not None: # only wake up the waiter when connection_made() has been called self._loop.call_soon(waiter._set_result_unless_cancelled, None) @@ -524,6 +544,15 @@ # write_eof is all what we needed to close the write pipe self.write_eof() + # On Python 3.3 and older, objects with a destructor part of a reference + # cycle are never destroyed. It's not more the case on Python 3.4 thanks + # to the PEP 442. + if sys.version_info >= (3, 4): + def __del__(self): + if self._pipe is not None: + warnings.warn("unclosed transport %r" % self, ResourceWarning) + self._pipe.close() + def abort(self): self._close(None) @@ -750,7 +779,7 @@ pass def add_child_handler(self, pid, callback, *args): - self._callbacks[pid] = callback, args + self._callbacks[pid] = (callback, args) # Prevent a race condition in case the child is already terminated. self._do_waitpid(pid) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -366,13 +366,16 @@ def _make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra=None, **kwargs): + waiter = futures.Future(loop=self) transp = _WindowsSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, - extra=extra, **kwargs) + waiter=waiter, extra=extra, + **kwargs) try: - yield from transp._post_init() + yield from waiter except: transp.close() + yield from transp._wait() raise return transp diff --git a/Lib/asyncio/windows_utils.py b/Lib/asyncio/windows_utils.py --- a/Lib/asyncio/windows_utils.py +++ b/Lib/asyncio/windows_utils.py @@ -14,6 +14,7 @@ import socket import subprocess import tempfile +import warnings __all__ = ['socketpair', 'pipe', 'Popen', 'PIPE', 'PipeHandle'] @@ -156,7 +157,10 @@ CloseHandle(self._handle) self._handle = None - __del__ = close + def __del__(self): + if self._handle is not None: + warnings.warn("unclosed %r" % self, ResourceWarning) + self.close() def __enter__(self): return self diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -886,13 +886,18 @@ if hasattr(sslcontext_client, 'check_hostname'): sslcontext_client.check_hostname = True + # no CA loaded f_c = self.loop.create_connection(MyProto, host, port, ssl=sslcontext_client) - with test_utils.disable_logger(): - with self.assertRaisesRegex(ssl.SSLError, - 'certificate verify failed '): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.SSLError, + 'certificate verify failed '): + self.loop.run_until_complete(f_c) + + # execute the loop to log the connection error + test_utils.run_briefly(self.loop) # close connection self.assertIsNone(proto.transport) @@ -919,15 +924,20 @@ f_c = self.loop.create_unix_connection(MyProto, path, ssl=sslcontext_client, server_hostname='invalid') - with test_utils.disable_logger(): - with self.assertRaisesRegex(ssl.SSLError, - 'certificate verify failed '): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex(ssl.SSLError, + 'certificate verify failed '): + self.loop.run_until_complete(f_c) + + # execute the loop to log the connection error + test_utils.run_briefly(self.loop) # close connection self.assertIsNone(proto.transport) server.close() + def test_legacy_create_unix_server_ssl_verify_failed(self): with test_utils.force_legacy_ssl_support(): self.test_create_unix_server_ssl_verify_failed() @@ -949,11 +959,12 @@ # incorrect server_hostname f_c = self.loop.create_connection(MyProto, host, port, ssl=sslcontext_client) - with test_utils.disable_logger(): - with self.assertRaisesRegex( - ssl.CertificateError, - "hostname '127.0.0.1' doesn't match 'localhost'"): - self.loop.run_until_complete(f_c) + with mock.patch.object(self.loop, 'call_exception_handler'): + with test_utils.disable_logger(): + with self.assertRaisesRegex( + ssl.CertificateError, + "hostname '127.0.0.1' doesn't match 'localhost'"): + self.loop.run_until_complete(f_c) # close connection proto.transport.close() @@ -1540,9 +1551,10 @@ stdin = transp.get_pipe_transport(0) stdin.write(b'Python The Winner') self.loop.run_until_complete(proto.got_data[1].wait()) - transp.close() + with test_utils.disable_logger(): + transp.close() self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) self.assertEqual(b'Python The Winner', proto.data[1]) def test_subprocess_interactive(self): @@ -1556,21 +1568,20 @@ self.loop.run_until_complete(proto.connected) self.assertEqual('CONNECTED', proto.state) - try: - stdin = transp.get_pipe_transport(0) - stdin.write(b'Python ') - self.loop.run_until_complete(proto.got_data[1].wait()) - proto.got_data[1].clear() - self.assertEqual(b'Python ', proto.data[1]) - - stdin.write(b'The Winner') - self.loop.run_until_complete(proto.got_data[1].wait()) - self.assertEqual(b'Python The Winner', proto.data[1]) - finally: + stdin = transp.get_pipe_transport(0) + stdin.write(b'Python ') + self.loop.run_until_complete(proto.got_data[1].wait()) + proto.got_data[1].clear() + self.assertEqual(b'Python ', proto.data[1]) + + stdin.write(b'The Winner') + self.loop.run_until_complete(proto.got_data[1].wait()) + self.assertEqual(b'Python The Winner', proto.data[1]) + + with test_utils.disable_logger(): transp.close() - self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) def test_subprocess_shell(self): connect = self.loop.subprocess_shell( @@ -1728,9 +1739,10 @@ # GetLastError()==ERROR_INVALID_NAME on Windows!?! (Using # WriteFile() we get ERROR_BROKEN_PIPE as expected.) self.assertEqual(b'ERR:OSError', proto.data[2]) - transp.close() + with test_utils.disable_logger(): + transp.close() self.loop.run_until_complete(proto.completed) - self.check_terminated(proto.returncode) + self.check_killed(proto.returncode) def test_subprocess_wait_no_same_group(self): # start the new process in a new session diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -499,8 +499,12 @@ self.proactor.accept.assert_called_with(self.sock) def test_socketpair(self): + class EventLoop(BaseProactorEventLoop): + # override the destructor to not log a ResourceWarning + def __del__(self): + pass self.assertRaises( - NotImplementedError, BaseProactorEventLoop, self.proactor) + NotImplementedError, EventLoop, self.proactor) def test_make_socket_transport(self): tr = self.loop._make_socket_transport(self.sock, asyncio.Protocol()) diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py --- a/Lib/test/test_asyncio/test_selector_events.py +++ b/Lib/test/test_asyncio/test_selector_events.py @@ -1427,7 +1427,7 @@ self.assertFalse(tr.can_write_eof()) self.assertRaises(NotImplementedError, tr.write_eof) - def test_close(self): + def check_close(self): tr = self._make_one() tr.close() @@ -1439,6 +1439,19 @@ self.assertEqual(tr._conn_lost, 1) self.assertEqual(1, self.loop.remove_reader_count[1]) + test_utils.run_briefly(self.loop) + + def test_close(self): + self.check_close() + self.assertTrue(self.protocol.connection_made.called) + self.assertTrue(self.protocol.connection_lost.called) + + def test_close_not_connected(self): + self.sslsock.do_handshake.side_effect = ssl.SSLWantReadError + self.check_close() + self.assertFalse(self.protocol.connection_made.called) + self.assertFalse(self.protocol.connection_lost.called) + @unittest.skipIf(ssl is None, 'No SSL support') def test_server_hostname(self): self.ssl_transport(server_hostname='localhost') diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py --- a/Lib/test/test_asyncio/test_sslproto.py +++ b/Lib/test/test_asyncio/test_sslproto.py @@ -22,7 +22,9 @@ def ssl_protocol(self, waiter=None): sslcontext = test_utils.dummy_ssl_context() app_proto = asyncio.Protocol() - return sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter) + proto = sslproto.SSLProtocol(self.loop, app_proto, sslcontext, waiter) + self.addCleanup(proto._app_transport.close) + return proto def connection_made(self, ssl_proto, do_handshake=None): transport = mock.Mock() @@ -56,9 +58,6 @@ with test_utils.disable_logger(): self.loop.run_until_complete(handshake_fut) - # Close the transport - ssl_proto._app_transport.close() - def test_eof_received_waiter(self): waiter = asyncio.Future(loop=self.loop) ssl_proto = self.ssl_protocol(waiter) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -4,6 +4,7 @@ from unittest import mock import asyncio +from asyncio import base_subprocess from asyncio import subprocess from asyncio import test_utils try: @@ -23,6 +24,56 @@ 'data = sys.stdin.buffer.read()', 'sys.stdout.buffer.write(data)'))] +class TestSubprocessTransport(base_subprocess.BaseSubprocessTransport): + def _start(self, *args, **kwargs): + self._proc = mock.Mock() + self._proc.stdin = None + self._proc.stdout = None + self._proc.stderr = None + + +class SubprocessTransportTests(test_utils.TestCase): + def setUp(self): + self.loop = self.new_test_loop() + self.set_event_loop(self.loop) + + + def create_transport(self, waiter=None): + protocol = mock.Mock() + protocol.connection_made._is_coroutine = False + protocol.process_exited._is_coroutine = False + transport = TestSubprocessTransport( + self.loop, protocol, ['test'], False, + None, None, None, 0, waiter=waiter) + return (transport, protocol) + + def test_proc_exited(self): + waiter = asyncio.Future(loop=self.loop) + transport, protocol = self.create_transport(waiter) + transport._process_exited(6) + self.loop.run_until_complete(waiter) + + self.assertEqual(transport.get_returncode(), 6) + + self.assertTrue(protocol.connection_made.called) + self.assertTrue(protocol.process_exited.called) + self.assertTrue(protocol.connection_lost.called) + self.assertEqual(protocol.connection_lost.call_args[0], (None,)) + + self.assertFalse(transport._closed) + self.assertIsNone(transport._loop) + self.assertIsNone(transport._proc) + self.assertIsNone(transport._protocol) + + # methods must raise ProcessLookupError if the process exited + self.assertRaises(ProcessLookupError, + transport.send_signal, signal.SIGTERM) + self.assertRaises(ProcessLookupError, transport.terminate) + self.assertRaises(ProcessLookupError, transport.kill) + + transport.close() + + class SubprocessMixin: def test_stdin_stdout(self): diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -350,16 +350,13 @@ return transport def test_ctor(self): - tr = self.read_pipe_transport() + waiter = asyncio.Future(loop=self.loop) + tr = self.read_pipe_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.protocol.connection_made.assert_called_with(tr) self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.protocol.connection_made.assert_called_with(tr) - - def test_ctor_with_waiter(self): - fut = asyncio.Future(loop=self.loop) - tr = self.read_pipe_transport(waiter=fut) - test_utils.run_briefly(self.loop) - self.assertIsNone(fut.result()) + self.assertIsNone(waiter.result()) @mock.patch('os.read') def test__read_ready(self, m_read): @@ -502,17 +499,13 @@ return transport def test_ctor(self): - tr = self.write_pipe_transport() + waiter = asyncio.Future(loop=self.loop) + tr = self.write_pipe_transport(waiter=waiter) + self.loop.run_until_complete(waiter) + + self.protocol.connection_made.assert_called_with(tr) self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.protocol.connection_made.assert_called_with(tr) - - def test_ctor_with_waiter(self): - fut = asyncio.Future(loop=self.loop) - tr = self.write_pipe_transport(waiter=fut) - self.loop.assert_reader(5, tr._read_ready) - test_utils.run_briefly(self.loop) - self.assertEqual(None, fut.result()) + self.assertEqual(None, waiter.result()) def test_can_write_eof(self): tr = self.write_pipe_transport() diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -360,6 +360,27 @@ self.assertEqual(list(reversed(m)), aslist) self.assertEqual(list(reversed(m)), list(m[::-1])) + def test_issue22668(self): + a = array.array('H', [256, 256, 256, 256]) + x = memoryview(a) + m = x.cast('B') + b = m.cast('H') + c = b[0:2] + d = memoryview(b) + + del b + + self.assertEqual(c[0], 256) + self.assertEqual(d[0], 256) + self.assertEqual(c.format, "H") + self.assertEqual(d.format, "H") + + _ = m.cast('I') + self.assertEqual(c[0], 256) + self.assertEqual(d[0], 256) + self.assertEqual(c.format, "H") + self.assertEqual(d.format, "H") + # Variations on source objects for the buffer: bytes-like objects, then arrays # with itemsize > 1. diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -965,7 +965,7 @@ check(int(PyLong_BASE**2-1), vsize('') + 2*self.longdigit) check(int(PyLong_BASE**2), vsize('') + 3*self.longdigit) # memoryview - check(memoryview(b''), size('Pnin 2P2n2i5P 3cPn')) + check(memoryview(b''), size('Pnin 2P2n2i5P Pn')) # module check(unittest, size('PnPPP')) # None diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -1132,6 +1132,51 @@ return -1; } +Py_LOCAL_INLINE(char *) +get_native_fmtstr(const char *fmt) +{ + int at = 0; + + if (fmt[0] == '@') { + at = 1; + fmt++; + } + if (fmt[0] == '\0' || fmt[1] != '\0') { + return NULL; + } + +#define RETURN(s) do { return at ? "@" s : s; } while (0) + + switch (fmt[0]) { + case 'c': RETURN("c"); + case 'b': RETURN("b"); + case 'B': RETURN("B"); + case 'h': RETURN("h"); + case 'H': RETURN("H"); + case 'i': RETURN("i"); + case 'I': RETURN("I"); + case 'l': RETURN("l"); + case 'L': RETURN("L"); + #ifdef HAVE_LONG_LONG + case 'q': RETURN("q"); + case 'Q': RETURN("Q"); + #endif + case 'n': RETURN("n"); + case 'N': RETURN("N"); + case 'f': RETURN("f"); + case 'd': RETURN("d"); + #ifdef HAVE_C99_BOOL + case '?': RETURN("?"); + #else + case '?': RETURN("?"); + #endif + case 'P': RETURN("P"); + } + + return NULL; +} + + /* Cast a memoryview's data type to 'format'. The input array must be C-contiguous. At least one of input-format, output-format must have byte size. The output array is 1-D, with the same byte length as the @@ -1181,10 +1226,13 @@ goto out; } - strncpy(mv->format, PyBytes_AS_STRING(asciifmt), - _Py_MEMORYVIEW_MAX_FORMAT); - mv->format[_Py_MEMORYVIEW_MAX_FORMAT-1] = '\0'; - view->format = mv->format; + view->format = get_native_fmtstr(PyBytes_AS_STRING(asciifmt)); + if (view->format == NULL) { + /* NOT_REACHED: get_native_fmtchar() already validates the format. */ + PyErr_SetString(PyExc_RuntimeError, + "memoryview: internal error"); + goto out; + } view->itemsize = itemsize; view->ndim = 1; -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Fri Jan 30 17:06:05 2015 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 30 Jan 2015 17:06:05 +0100 Subject: [Python-checkins] Daily reference leaks (efb25925bd13): sum=1749067 Message-ID: results for efb25925bd13 on branch "default" -------------------------------------------- test_asyncio leaked [479454, 479295, 487404] references, sum=1446153 test_asyncio leaked [100253, 100217, 102444] memory blocks, sum=302914 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogD_apOC', '-x'] From python-checkins at python.org Fri Jan 30 19:33:50 2015 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 30 Jan 2015 18:33:50 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_allow_changing_=5F=5Fclass?= =?utf-8?q?=5F=5F_between_a_heaptype_and_non-heaptype_in_some_cases?= Message-ID: <20150130183346.34410.72039@psf.io> https://hg.python.org/cpython/rev/c0d25de5919e changeset: 94397:c0d25de5919e user: Benjamin Peterson date: Fri Jan 30 13:33:42 2015 -0500 summary: allow changing __class__ between a heaptype and non-heaptype in some cases (closes #22986) Patch by Nathaniel Smith. files: Lib/test/test_descr.py | 16 +++++++ Misc/NEWS | 3 + Objects/typeobject.c | 65 +++++++++++++++++------------ 3 files changed, 57 insertions(+), 27 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -1026,6 +1026,22 @@ self.assertEqual(x.foo, 1) self.assertEqual(x.__dict__, {'foo': 1}) + def test_object_class_assignment_between_heaptypes_and_nonheaptypes(self): + class SubType(types.ModuleType): + a = 1 + + m = types.ModuleType("m") + self.assertTrue(m.__class__ is types.ModuleType) + self.assertFalse(hasattr(m, "a")) + + m.__class__ = SubType + self.assertTrue(m.__class__ is SubType) + self.assertTrue(hasattr(m, "a")) + + m.__class__ = types.ModuleType + self.assertTrue(m.__class__ is types.ModuleType) + self.assertFalse(hasattr(m, "a")) + def test_slots(self): # Testing __slots__... class C0(object): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #22986: Allow changing an object's __class__ between a dynamic type and + static type in some cases. + - Issue #15859: PyUnicode_EncodeFSDefault(), PyUnicode_EncodeMBCS() and PyUnicode_EncodeCodePage() now raise an exception if the object is not an Unicode object. For PyUnicode_EncodeFSDefault(), it was already the case on diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -1196,8 +1196,11 @@ assert(basedealloc); basedealloc(self); - /* Can't reference self beyond this point */ - Py_DECREF(type); + /* Can't reference self beyond this point. It's possible tp_del switched + our type from a HEAPTYPE to a non-HEAPTYPE, so be careful about + reference counting. */ + if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) + Py_DECREF(type); endlabel: ++_PyTrash_delete_nesting; @@ -3425,17 +3428,18 @@ } static int -equiv_structs(PyTypeObject *a, PyTypeObject *b) -{ - return a == b || - (a != NULL && - b != NULL && - a->tp_basicsize == b->tp_basicsize && - a->tp_itemsize == b->tp_itemsize && - a->tp_dictoffset == b->tp_dictoffset && - a->tp_weaklistoffset == b->tp_weaklistoffset && - ((a->tp_flags & Py_TPFLAGS_HAVE_GC) == - (b->tp_flags & Py_TPFLAGS_HAVE_GC))); +compatible_with_tp_base(PyTypeObject *child) +{ + PyTypeObject *parent = child->tp_base; + return (parent != NULL && + child->tp_basicsize == parent->tp_basicsize && + child->tp_itemsize == parent->tp_itemsize && + child->tp_dictoffset == parent->tp_dictoffset && + child->tp_weaklistoffset == parent->tp_weaklistoffset && + ((child->tp_flags & Py_TPFLAGS_HAVE_GC) == + (parent->tp_flags & Py_TPFLAGS_HAVE_GC)) && + (child->tp_dealloc == subtype_dealloc || + child->tp_dealloc == parent->tp_dealloc)); } static int @@ -3453,6 +3457,10 @@ size += sizeof(PyObject *); /* Check slots compliance */ + if (!(a->tp_flags & Py_TPFLAGS_HEAPTYPE) || + !(b->tp_flags & Py_TPFLAGS_HEAPTYPE)) { + return 0; + } slots_a = ((PyHeapTypeObject *)a)->ht_slots; slots_b = ((PyHeapTypeObject *)b)->ht_slots; if (slots_a && slots_b) { @@ -3468,9 +3476,7 @@ { PyTypeObject *newbase, *oldbase; - if (newto->tp_dealloc != oldto->tp_dealloc || - newto->tp_free != oldto->tp_free) - { + if (newto->tp_free != oldto->tp_free) { PyErr_Format(PyExc_TypeError, "%s assignment: " "'%s' deallocator differs from '%s'", @@ -3479,11 +3485,21 @@ oldto->tp_name); return 0; } + /* + It's tricky to tell if two arbitrary types are sufficiently compatible as + to be interchangeable; e.g., even if they have the same tp_basicsize, they + might have totally different struct fields. It's much easier to tell if a + type and its supertype are compatible; e.g., if they have the same + tp_basicsize, then that means they have identical fields. So to check + whether two arbitrary types are compatible, we first find the highest + supertype that each is compatible with, and then if those supertypes are + compatible then the original types must also be compatible. + */ newbase = newto; oldbase = oldto; - while (equiv_structs(newbase, newbase->tp_base)) + while (compatible_with_tp_base(newbase)) newbase = newbase->tp_base; - while (equiv_structs(oldbase, oldbase->tp_base)) + while (compatible_with_tp_base(oldbase)) oldbase = oldbase->tp_base; if (newbase != oldbase && (newbase->tp_base != oldbase->tp_base || @@ -3518,17 +3534,12 @@ return -1; } newto = (PyTypeObject *)value; - if (!(newto->tp_flags & Py_TPFLAGS_HEAPTYPE) || - !(oldto->tp_flags & Py_TPFLAGS_HEAPTYPE)) - { - PyErr_Format(PyExc_TypeError, - "__class__ assignment: only for heap types"); - return -1; - } if (compatible_for_assignment(oldto, newto, "__class__")) { - Py_INCREF(newto); + if (newto->tp_flags & Py_TPFLAGS_HEAPTYPE) + Py_INCREF(newto); Py_TYPE(self) = newto; - Py_DECREF(oldto); + if (oldto->tp_flags & Py_TPFLAGS_HEAPTYPE) + Py_DECREF(oldto); return 0; } else { -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 20:11:44 2015 From: python-checkins at python.org (stefan.krah) Date: Fri, 30 Jan 2015 19:11:44 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMzQ5?= =?utf-8?q?=3A_Fix_off-by-one_error_in_PyBuffer=5FToContiguous=28=29=2E_In?= =?utf-8?q?itial_patch?= Message-ID: <20150130191141.96078.54544@psf.io> https://hg.python.org/cpython/rev/a9305102c892 changeset: 94398:a9305102c892 branch: 2.7 parent: 94357:a18444102268 user: Stefan Krah date: Fri Jan 30 20:11:10 2015 +0100 summary: Issue #23349: Fix off-by-one error in PyBuffer_ToContiguous(). Initial patch by Richard Hansen. files: Misc/ACKS | 1 + Modules/_testcapimodule.c | 48 +++++++++++++++++++++++++++ Objects/abstract.c | 4 +- 3 files changed, 51 insertions(+), 2 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -518,6 +518,7 @@ Harald Hanche-Olsen Manus Hand Milton L. Hankins +Richard Hansen Stephen Hansen Barry Hantman Lynda Hardman diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -376,6 +376,53 @@ Py_RETURN_NONE; } +static PyObject * +test_to_contiguous(PyObject* self, PyObject *noargs) +{ + int data[9] = {0, -1, 1, -1, 2, -1, 3, -1, 4}; + int result[5]; + Py_ssize_t itemsize = sizeof(int); + Py_ssize_t shape = 5; + Py_ssize_t strides = 2 * itemsize; + Py_buffer view = { + data, + NULL, + 5 * itemsize, + itemsize, + 1, + 1, + NULL, + &shape, + &strides, + NULL, + {0, 0}, + NULL + }; + int i; + + PyBuffer_ToContiguous(result, &view, view.len, 'C'); + for (i = 0; i < 5; i++) { + if (result[i] != i) { + PyErr_SetString(TestError, + "test_to_contiguous: incorrect result"); + return NULL; + } + } + + view.buf = &data[8]; + view.strides[0] = -2 * itemsize; + + PyBuffer_ToContiguous(result, &view, view.len, 'C'); + for (i = 0; i < 5; i++) { + if (result[i] != 4-i) { + PyErr_SetString(TestError, + "test_to_contiguous: incorrect result"); + return NULL; + } + } + + Py_RETURN_NONE; +} /* Tests of PyLong_{As, From}{Unsigned,}Long(), and (#ifdef HAVE_LONG_LONG) PyLong_{As, From}{Unsigned,}LongLong(). @@ -1785,6 +1832,7 @@ {"test_dict_iteration", (PyCFunction)test_dict_iteration,METH_NOARGS}, {"test_lazy_hash_inheritance", (PyCFunction)test_lazy_hash_inheritance,METH_NOARGS}, {"test_broken_memoryview", (PyCFunction)test_broken_memoryview,METH_NOARGS}, + {"test_to_contiguous", (PyCFunction)test_to_contiguous, METH_NOARGS}, {"test_long_api", (PyCFunction)test_long_api, METH_NOARGS}, {"test_long_and_overflow", (PyCFunction)test_long_and_overflow, METH_NOARGS}, diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -499,7 +499,7 @@ /* Otherwise a more elaborate scheme is needed */ - /* XXX(nnorwitz): need to check for overflow! */ + /* view->ndim <= 64 */ indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim)); if (indices == NULL) { PyErr_NoMemory(); @@ -521,10 +521,10 @@ */ elements = len / view->itemsize; while (elements--) { - addone(view->ndim, indices, view->shape); ptr = PyBuffer_GetPointer(view, indices); memcpy(dest, ptr, view->itemsize); dest += view->itemsize; + addone(view->ndim, indices, view->shape); } PyMem_Free(indices); return 0; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Jan 30 21:02:29 2015 From: python-checkins at python.org (berker.peksag) Date: Fri, 30 Jan 2015 20:02:29 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_448=3A_Updates_from_Neil_?= =?utf-8?q?Girdhar=2E?= Message-ID: <20150130200214.25865.50298@psf.io> https://hg.python.org/peps/rev/26c3392051e6 changeset: 5682:26c3392051e6 user: Berker Peksag date: Fri Jan 30 22:02:45 2015 +0200 summary: PEP 448: Updates from Neil Girdhar. files: pep-0448.txt | 113 ++++++++++++++++---------------------- 1 files changed, 48 insertions(+), 65 deletions(-) diff --git a/pep-0448.txt b/pep-0448.txt --- a/pep-0448.txt +++ b/pep-0448.txt @@ -16,30 +16,22 @@ ======== This PEP proposes extended usages of the ``*`` iterable unpacking -operator to allow unpacking in more positions, an arbitrary number of -times, and in several additional circumstances. +operator and ``**`` dictionary unpacking operator +to allow unpacking in more positions, an arbitrary number of +times, and in additional circumstances. Specifically +in function calls, in comprehensions and generator expressions, +and in displays. -Specifically: - -Arbitrarily positioned unpacking operators:: +Function calls are proposed to support an arbitrary number of +unpackings rather than just one:: >>> print(*[1], *[2], 3) 1 2 3 >>> dict(**{'x': 1}, y=2, **{'z': 3}) {'x': 1, 'y': 2, 'z': 3} -Function calls currently have the restriction that keyword arguments -must follow positional arguments and ``**`` unpackings must additionally -follow ``*`` unpackings. Because of the new levity for ``*`` and ``**`` -unpackings, it may be advisable to lift some or all of these -restrictions. - -As currently, if an argument is given multiple times - such as a -positional argument given both positionally and by keyword - a -TypeError is raised. - -Unpacking is proposed to be allowed inside tuples, lists, sets, -dictionaries and comprehensions:: +Unpacking is proposed to be allowed inside tuple, list, set, +and dictionary displays:: >>> *range(4), 4 (0, 1, 2, 3, 4) @@ -50,10 +42,25 @@ >>> {'x': 1, **{'y': 2}} {'x': 1, 'y': 2} +In sets and dictionaries, this provides a concise overriding +notation:: + + >>> {'x': 1, **{'x': 2}} + {'x': 2} + + >>> {**{'x': 2}, 'x': 1} + {'x': 1} + +Unpacking is proposed to be allowed inside list, set, +and dictionary comprehensions:: + >>> ranges = [range(i) for i in range(5)] >>> [*item for item in ranges] [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] + >>> {*item for item in ranges} + {0, 1, 2, 3} + Rationale ========= @@ -124,38 +131,29 @@ Specification ============= -Function calls may accept an unbound number of ``*`` and ``**`` +Function calls may accept an unbounded number of ``*`` and ``**`` unpackings. There will be no restriction of the order of positional arguments with relation to ``*`` unpackings nor any restriction of the order of keyword arguments with relation to ``**`` unpackings. Function calls currently have the restriction that keyword arguments must follow positional arguments and ``**`` unpackings must additionally -follow ``*`` unpackings. Because of the new levity for ``*`` and ``**`` -unpackings, it may be advisable to list some or all of these -restrictions. +follow ``*`` unpackings. -As currently, if an argument is given multiple times - such as a -positional argument given both positionally and by keyword - a -TypeError is raised. +Currently, if an argument is given multiple times ? such as a +positional argument given both positionally and by keyword ? a +``TypeError`` is raised. This remains true for duplicate arguments +provided through multiple keyword argument unpackings, +e.g. ``f(**{'x': 2}, **{'x': 3})``. -If the restrictions are kept, a function call will look like this:: +A function looks like this:: - function( - argument or *args, argument or *args, ..., - kwargument or *args, kwargument or *args, ..., - kwargument or **kwargs, kwargument or **kwargs, ... - ) - -If they are removed completely, a function call will look like this:: - function( - argument or keyword_argument or *args or **kwargs, - argument or keyword_argument or *args or **kwargs, - ... + argument or *args, argument or *args, ..., + kwargument or *args, kwargument or *args, ..., + kwargument or **kwargs, kwargument or **kwargs, ... ) - Tuples, lists, sets and dictionaries will allow unpacking. This will act as if the elements from unpacked items were inserted in order at the site of unpacking, much as happens in unpacking in a function-call. @@ -168,7 +166,7 @@ dictionaries require ``**`` unpacking, all the others require ``*`` unpacking and key priorities are unchanged. -Examples include:: +For example:: {*[1, 2, 3], 4, 5, *{6, 7, 8}} @@ -178,45 +176,30 @@ {**locals(), "override": None} +However, ``f(*x for x in it)`` and ``f(**x for x in it)`` continue +to raise SyntaxError. + Disadvantages ============= -If the current restrictions for function call arguments (keyword -arguments must follow positional arguments and ``**`` unpackings must -additionally follow ``*`` unpackings) are kept, the allowable orders -for arguments in a function call is more complicated than before. +The allowable orders for arguments in a function call is more +complicated than before. The simplest explanation for the rules may be "positional arguments -come first and keyword arguments follow, but ``*`` unpackings are -allowed after keyword arguments". +come first and keyword arguments follow, but iterable unpackings are +allowed after keyword arguments" or "iterable arguments precede +all keyword arguments and keyword argument unpackings, and iterable +argument unpackings precede all keyword argument unpackings". -If the current restrictions are lifted, there are no obvious gains to -code as the only new orders that are allowed look silly: ``f(a, e=e, -d=d, b, c)`` being a simpler example. - - -Whilst ``*elements, = iterable`` causes ``elements`` to be a list, +While ``*elements, = iterable`` causes ``elements`` to be a list, ``elements = *iterable,`` causes ``elements`` to be a tuple. The -reason for this may not be obvious at first glance and may confuse -people unfamiliar with the construct. +reason for this may confuse people unfamiliar with the construct. Implementation ============== -An implementation for an old version of Python 3 is found at Issue -2292 on bug tracker [1]_, although several changes should be made: - -- It has yet to be updated to the most recent Python version - -- It features a now redundant replacement for "yield from" which - should be removed - -- It also loses support for calling function with keyword arguments before - positional arguments, which is an unnecessary backwards-incompatible change - -- If the restrictions on the order of arguments in a function call are - partially or fully lifted, they would need to be included +An implementation for Python 3.5 is found at Issue 2292 on bug tracker [1]_. References -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Jan 30 22:36:08 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 30 Jan 2015 21:36:08 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMDU1?= =?utf-8?q?=3A_Fixed_off-by-one_error_in_PyUnicode=5FFromFormatV=2E?= Message-ID: <20150130213559.39274.28717@psf.io> https://hg.python.org/cpython/rev/e5d79e6deeb5 changeset: 94399:e5d79e6deeb5 branch: 2.7 user: Serhiy Storchaka date: Fri Jan 30 23:35:03 2015 +0200 summary: Issue #23055: Fixed off-by-one error in PyUnicode_FromFormatV. files: Objects/unicodeobject.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -893,7 +893,8 @@ } expand: if (abuffersize > 20) { - abuffer = PyObject_Malloc(abuffersize); + /* add 1 for sprintf's trailing null byte */ + abuffer = PyObject_Malloc(abuffersize + 1); if (!abuffer) { PyErr_NoMemory(); goto fail; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 00:17:07 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 30 Jan 2015 23:17:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzIzMDU1?= =?utf-8?q?=3A_Fixed_read-past-the-end_error_in_PyUnicode=5FFromFormatV=2E?= Message-ID: <20150130231707.106357.81443@psf.io> https://hg.python.org/cpython/rev/9fe1d861f486 changeset: 94401:9fe1d861f486 branch: 3.2 parent: 94339:f849f937f78c user: Serhiy Storchaka date: Sat Jan 31 01:15:48 2015 +0200 summary: Issue #23055: Fixed read-past-the-end error in PyUnicode_FromFormatV. files: Objects/unicodeobject.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -762,6 +762,8 @@ f++; while (*f && *f != '%' && !Py_ISALPHA((unsigned)*f)) f++; + if (!*f) + break; if (*f == 's' || *f=='S' || *f=='R' || *f=='A' || *f=='V') ++callcount; } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 00:17:07 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 30 Jan 2015 23:17:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMDU1?= =?utf-8?q?=3A_Fixed_read-past-the-end_error_in_PyUnicode=5FFromFormatV=2E?= Message-ID: <20150130231707.96094.4409@psf.io> https://hg.python.org/cpython/rev/245c9f372a34 changeset: 94400:245c9f372a34 branch: 2.7 user: Serhiy Storchaka date: Sat Jan 31 01:15:29 2015 +0200 summary: Issue #23055: Fixed read-past-the-end error in PyUnicode_FromFormatV. files: Objects/unicodeobject.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -738,6 +738,8 @@ f++; while (*f && *f != '%' && !isalpha((unsigned)*f)) f++; + if (!*f) + break; if (*f == 's' || *f=='S' || *f=='R') ++callcount; } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 01:13:06 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 00:13:06 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Use_float_divi?= =?utf-8?q?sion_to_avoid_deprecation_warning_in_test=5Ftimeit_=28issue_=23?= =?utf-8?q?11578=29=2E?= Message-ID: <20150131001304.96084.91812@psf.io> https://hg.python.org/cpython/rev/aa6f8e067ec3 changeset: 94402:aa6f8e067ec3 branch: 2.7 parent: 94400:245c9f372a34 user: Serhiy Storchaka date: Sat Jan 31 02:12:17 2015 +0200 summary: Use float division to avoid deprecation warning in test_timeit (issue #11578). files: Lib/test/test_timeit.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_timeit.py b/Lib/test/test_timeit.py --- a/Lib/test/test_timeit.py +++ b/Lib/test/test_timeit.py @@ -296,12 +296,12 @@ def test_main_exception(self): with captured_stderr() as error_stringio: - s = self.run_main(switches=['1/0']) + s = self.run_main(switches=['1.0/0.0']) self.assert_exc_string(error_stringio.getvalue(), 'ZeroDivisionError') def test_main_exception_fixed_reps(self): with captured_stderr() as error_stringio: - s = self.run_main(switches=['-n1', '1/0']) + s = self.run_main(switches=['-n1', '1.0/0.0']) self.assert_exc_string(error_stringio.getvalue(), 'ZeroDivisionError') -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 03:02:26 2015 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 31 Jan 2015 02:02:26 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_typo_in_a_comment=2E?= Message-ID: <20150131020221.106264.71540@psf.io> https://hg.python.org/cpython/rev/73c31db9fba4 changeset: 94403:73c31db9fba4 parent: 94397:c0d25de5919e user: Raymond Hettinger date: Fri Jan 30 18:02:15 2015 -0800 summary: Fix typo in a comment. files: Objects/setobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -681,7 +681,7 @@ hash *= (Py_uhash_t)PySet_GET_SIZE(self) + 1; while (set_next(so, &pos, &entry)) { /* Work to increase the bit dispersion for closely spaced hash - values. The is important because some use cases have many + values. This is important because some use cases have many combinations of a small number of elements with nearby hashes so that many distinct combinations collapse to only a handful of distinct hash values. */ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 05:10:14 2015 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 31 Jan 2015 04:10:14 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Minor_tweak_to_improve_cod?= =?utf-8?q?e_clarity=2E?= Message-ID: <20150131040929.96084.9253@psf.io> https://hg.python.org/cpython/rev/f726b27bda8d changeset: 94404:f726b27bda8d user: Raymond Hettinger date: Fri Jan 30 20:09:23 2015 -0800 summary: Minor tweak to improve code clarity. files: Objects/setobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -150,7 +150,7 @@ goto found_null; if (i + LINEAR_PROBES <= mask) { for (j = 1; j <= LINEAR_PROBES; j++) { - entry = &table[i + j]; + entry++; if (entry->key == NULL) goto found_null; } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 08:26:15 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 07:26:15 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Restored_test?= =?utf-8?q?=5Fxpickle_compatibility_with_Python_2=2E5=2E?= Message-ID: <20150131072615.25863.16056@psf.io> https://hg.python.org/cpython/rev/19305fa6420b changeset: 94405:19305fa6420b branch: 2.7 parent: 94402:aa6f8e067ec3 user: Serhiy Storchaka date: Sat Jan 31 09:25:16 2015 +0200 summary: Restored test_xpickle compatibility with Python 2.5. Python 2.5 has no unittest.skipUnless. files: Lib/test/test_xpickle.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_xpickle.py b/Lib/test/test_xpickle.py --- a/Lib/test/test_xpickle.py +++ b/Lib/test/test_xpickle.py @@ -158,8 +158,10 @@ # This is a cut-down version of pickletester's test_unicode. Backwards # compatibility was explicitly broken in r67934 to fix a bug. - @unittest.skipUnless(test_support.have_unicode, 'no unicode support') def test_unicode(self): + if not test_support.have_unicode: + # Python 2.5 has no unittest.skipUnless + self.skipTest('no unicode support') endcases = [u'', u'<\\u>', u'<\\%c>' % 0x1234, u'<\n>', u'<\\>'] for proto in pickletester.protocols: for u in endcases: @@ -218,7 +220,7 @@ python = "python2.7" error = cPickle.BadPickleGet -class PicklePython27Compat(CPicklePython26Compat): +class PicklePython27Compat(CPicklePython27Compat): module = pickle error = KeyError -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 09:38:50 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 08:38:50 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogdGVzdF9zcHdkIHNr?= =?utf-8?q?ipping_is_expected_on_win32=2E?= Message-ID: <20150131083846.34384.12418@psf.io> https://hg.python.org/cpython/rev/f5df01f69a3e changeset: 94406:f5df01f69a3e branch: 2.7 user: Serhiy Storchaka date: Sat Jan 31 10:20:31 2015 +0200 summary: test_spwd skipping is expected on win32. files: Lib/test/regrtest.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -1191,6 +1191,7 @@ test_pwd test_resource test_signal + test_spwd test_threadsignals test_timing test_wait3 -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 10:27:59 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 09:27:59 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Avoid_deprecat?= =?utf-8?q?ion_warnings=2E?= Message-ID: <20150131092748.25845.71271@psf.io> https://hg.python.org/cpython/rev/6ddcc1407ffa changeset: 94407:6ddcc1407ffa branch: 2.7 user: Serhiy Storchaka date: Sat Jan 31 11:27:06 2015 +0200 summary: Avoid deprecation warnings. files: Lib/ctypes/util.py | 2 +- Lib/encodings/uu_codec.py | 2 +- Lib/test/test_descr.py | 2 +- Lib/test/test_exceptions.py | 2 +- Lib/test/test_ssl.py | 2 +- Lib/test/test_threading.py | 6 +++--- Lib/test/test_timeit.py | 2 +- 7 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Lib/ctypes/util.py b/Lib/ctypes/util.py --- a/Lib/ctypes/util.py +++ b/Lib/ctypes/util.py @@ -177,7 +177,7 @@ res = re.findall(expr, data) if not res: return _get_soname(_findLib_gcc(name)) - res.sort(cmp= lambda x,y: cmp(_num_version(x), _num_version(y))) + res.sort(key=_num_version) return res[-1] elif sys.platform == "sunos5": diff --git a/Lib/encodings/uu_codec.py b/Lib/encodings/uu_codec.py --- a/Lib/encodings/uu_codec.py +++ b/Lib/encodings/uu_codec.py @@ -84,7 +84,7 @@ data = a2b_uu(s) except binascii.Error, v: # Workaround for broken uuencoders by /Fredrik Lundh - nbytes = (((ord(s[0])-32) & 63) * 4 + 5) / 3 + nbytes = (((ord(s[0])-32) & 63) * 4 + 5) // 3 data = a2b_uu(s[:nbytes]) #sys.stderr.write("Warning: %s\n" % str(v)) write(data) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -2066,7 +2066,7 @@ "attr on a property" % attr) class D(object): - __getitem__ = property(lambda s: 1/0) + __getitem__ = property(lambda s: 1.0/0.0) d = D() try: diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -92,7 +92,7 @@ self.raise_catch(TabError, "TabError") # can only be tested under -tt, and is the only test for -tt - #try: compile("try:\n\t1/0\n \t1/0\nfinally:\n pass\n", '', 'exec') + #try: compile("try:\n\t1.0/0.0\n \t1.0/0.0\nfinally:\n pass\n", '', 'exec') #except TabError: pass #else: self.fail("TabError not raised") diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2954,7 +2954,7 @@ server_context, other_context, client_context = self.sni_contexts() def cb_raising(ssl_sock, server_name, initial_context): - 1/0 + 1.0/0.0 server_context.set_servername_callback(cb_raising) with self.assertRaises(ssl.SSLError) as cm, \ diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -797,7 +797,7 @@ running = True while running: time.sleep(0.01) - 1/0 + 1.0/0.0 t = threading.Thread(target=run) t.start() while not running: @@ -824,7 +824,7 @@ running = True while running: time.sleep(0.01) - 1/0 + 1.0/0.0 t = threading.Thread(target=run) t.start() while not running: @@ -852,7 +852,7 @@ running = True while running: time.sleep(0.01) - 1/0 + 1.0/0.0 sys.stderr = None t = threading.Thread(target=run) t.start() diff --git a/Lib/test/test_timeit.py b/Lib/test/test_timeit.py --- a/Lib/test/test_timeit.py +++ b/Lib/test/test_timeit.py @@ -197,7 +197,7 @@ def test_print_exc(self): s = StringIO() - t = timeit.Timer("1/0") + t = timeit.Timer("1.0/0.0") try: t.timeit() except: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 10:51:07 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 09:51:07 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Completed_Misc?= =?utf-8?q?/NEWS_entry=2E?= Message-ID: <20150131095107.39290.77151@psf.io> https://hg.python.org/cpython/rev/db80b19730f0 changeset: 94408:db80b19730f0 branch: 2.7 user: Serhiy Storchaka date: Sat Jan 31 11:42:04 2015 +0200 summary: Completed Misc/NEWS entry. files: Misc/NEWS | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -86,7 +86,8 @@ Tests ----- -- Issue #19949: The test_xpickle test now +- Issue #19949: The test_xpickle test now tests compatibility with installed + Python 2.7 and reports skipped tests. Based on patch by Zachary Ware. - Issue #11578: Backported test for the timeit module. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 10:51:07 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 09:51:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIyNzY1?= =?utf-8?q?=3A_Fixed_test=5Fgdb_failures=2E_Supressed_unexpected_gdb_outpu?= =?utf-8?q?t=2E?= Message-ID: <20150131095107.39294.16179@psf.io> https://hg.python.org/cpython/rev/47a9cb7ec0cb changeset: 94409:47a9cb7ec0cb branch: 2.7 user: Serhiy Storchaka date: Sat Jan 31 11:48:36 2015 +0200 summary: Issue #22765: Fixed test_gdb failures. Supressed unexpected gdb output. Patch by Bohuslav Kabrda. files: Lib/test/test_gdb.py | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -118,6 +118,25 @@ # Generate a list of commands in gdb's language: commands = ['set breakpoint pending yes', 'break %s' % breakpoint, + + # GDB as of 7.4 (?) onwards can distinguish between the + # value of a variable at entry vs current value: + # http://sourceware.org/gdb/onlinedocs/gdb/Variables.html + # which leads to the selftests failing with errors like this: + # AssertionError: 'v at entry=()' != '()' + # Disable this: + 'set print entry-values no', + + # The tests assume that the first frame of printed + # backtrace will not contain program counter, + # that is however not guaranteed by gdb + # therefore we need to use 'set print address off' to + # make sure the counter is not there. For example: + # #0 in PyObject_Print ... + # is assumed, but sometimes this can be e.g. + # #0 0x00003fffb7dd1798 in PyObject_Print ... + 'set print address off', + 'run'] if cmds_after_breakpoint: commands += cmds_after_breakpoint -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 10:51:07 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 09:51:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNzY1?= =?utf-8?q?=3A_Fixed_test=5Fgdb_failures=2E_Supressed_unexpected_gdb_outpu?= =?utf-8?q?t=2E?= Message-ID: <20150131095107.34394.17240@psf.io> https://hg.python.org/cpython/rev/5b5a581d91c8 changeset: 94410:5b5a581d91c8 branch: 3.4 parent: 94393:fadfcb2d5a47 user: Serhiy Storchaka date: Sat Jan 31 11:48:52 2015 +0200 summary: Issue #22765: Fixed test_gdb failures. Supressed unexpected gdb output. Patch by Bohuslav Kabrda. files: Lib/test/test_gdb.py | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -123,6 +123,25 @@ # Generate a list of commands in gdb's language: commands = ['set breakpoint pending yes', 'break %s' % breakpoint, + + # GDB as of 7.4 (?) onwards can distinguish between the + # value of a variable at entry vs current value: + # http://sourceware.org/gdb/onlinedocs/gdb/Variables.html + # which leads to the selftests failing with errors like this: + # AssertionError: 'v at entry=()' != '()' + # Disable this: + 'set print entry-values no', + + # The tests assume that the first frame of printed + # backtrace will not contain program counter, + # that is however not guaranteed by gdb + # therefore we need to use 'set print address off' to + # make sure the counter is not there. For example: + # #0 in PyObject_Print ... + # is assumed, but sometimes this can be e.g. + # #0 0x00003fffb7dd1798 in PyObject_Print ... + 'set print address off', + 'run'] if cmds_after_breakpoint: commands += cmds_after_breakpoint -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 10:51:07 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 09:51:07 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322765=3A_Fixed_test=5Fgdb_failures=2E_Supressed?= =?utf-8?q?_unexpected_gdb_output=2E?= Message-ID: <20150131095107.106317.92467@psf.io> https://hg.python.org/cpython/rev/3813a5282eac changeset: 94411:3813a5282eac parent: 94404:f726b27bda8d parent: 94410:5b5a581d91c8 user: Serhiy Storchaka date: Sat Jan 31 11:50:22 2015 +0200 summary: Issue #22765: Fixed test_gdb failures. Supressed unexpected gdb output. Patch by Bohuslav Kabrda. files: Lib/test/test_gdb.py | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -123,6 +123,25 @@ # Generate a list of commands in gdb's language: commands = ['set breakpoint pending yes', 'break %s' % breakpoint, + + # GDB as of 7.4 (?) onwards can distinguish between the + # value of a variable at entry vs current value: + # http://sourceware.org/gdb/onlinedocs/gdb/Variables.html + # which leads to the selftests failing with errors like this: + # AssertionError: 'v at entry=()' != '()' + # Disable this: + 'set print entry-values no', + + # The tests assume that the first frame of printed + # backtrace will not contain program counter, + # that is however not guaranteed by gdb + # therefore we need to use 'set print address off' to + # make sure the counter is not there. For example: + # #0 in PyObject_Print ... + # is assumed, but sometimes this can be e.g. + # #0 0x00003fffb7dd1798 in PyObject_Print ... + 'set print address off', + 'run'] if cmds_after_breakpoint: commands += cmds_after_breakpoint -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 11:09:06 2015 From: python-checkins at python.org (victor.stinner) Date: Sat, 31 Jan 2015 10:09:06 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40IChnZW5lcmF0b3Ip?= Message-ID: <20150131100902.39282.14095@psf.io> https://hg.python.org/cpython/rev/cb5cdb0aa18d changeset: 94413:cb5cdb0aa18d parent: 94411:3813a5282eac parent: 94412:4555da8c3091 user: Victor Stinner date: Sat Jan 31 11:08:40 2015 +0100 summary: Merge 3.4 (generator) files: Lib/test/test_generators.py | 109 ++++++++++++++++++++++++ Misc/NEWS | 6 + Python/ceval.c | 7 +- 3 files changed, 119 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -89,6 +89,115 @@ "GeneratorTest.test_name..") +class ExceptionTest(unittest.TestCase): + # Tests for the issue #23353: check that the currently handled exception + # is correctly saved/restored in PyEval_EvalFrameEx(). + + def test_except_throw(self): + def store_raise_exc_generator(): + try: + self.assertEqual(sys.exc_info()[0], None) + yield + except Exception as exc: + # exception raised by gen.throw(exc) + self.assertEqual(sys.exc_info()[0], ValueError) + self.assertIsNone(exc.__context__) + yield + + # ensure that the exception is not lost + self.assertEqual(sys.exc_info()[0], ValueError) + yield + + # we should be able to raise back the ValueError + raise + + make = store_raise_exc_generator() + next(make) + + try: + raise ValueError() + except Exception as exc: + try: + make.throw(exc) + except Exception: + pass + + next(make) + with self.assertRaises(ValueError) as cm: + next(make) + self.assertIsNone(cm.exception.__context__) + + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_next(self): + def gen(): + self.assertEqual(sys.exc_info()[0], ValueError) + yield "done" + + g = gen() + try: + raise ValueError + except Exception: + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_gen_except(self): + def gen(): + try: + self.assertEqual(sys.exc_info()[0], None) + yield + # we are called from "except ValueError:", TypeError must + # inherit ValueError in its context + raise TypeError() + except TypeError as exc: + self.assertEqual(sys.exc_info()[0], TypeError) + self.assertEqual(type(exc.__context__), ValueError) + # here we are still called from the "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + yield + self.assertIsNone(sys.exc_info()[0]) + yield "done" + + g = gen() + next(g) + try: + raise ValueError + except Exception: + next(g) + + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_throw_exception_context(self): + def gen(): + try: + try: + self.assertEqual(sys.exc_info()[0], None) + yield + except ValueError: + # we are called from "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + raise TypeError() + except Exception as exc: + self.assertEqual(sys.exc_info()[0], TypeError) + self.assertEqual(type(exc.__context__), ValueError) + # we are still called from "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + yield + self.assertIsNone(sys.exc_info()[0]) + yield "done" + + g = gen() + next(g) + try: + raise ValueError + except Exception as exc: + g.throw(exc) + + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + tutorial_tests = """ Let's try a simple generator: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -226,6 +226,12 @@ Library ------- +- Issue #23353: Fix the exception handling of generators in + PyEval_EvalFrameEx(). At entry, save or swap the exception state even if + PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception state + is now always restored or swapped, not only if why is WHY_YIELD or + WHY_RETURN. Patch co-written with Antoine Pitrou. + - Issue #14099: Restored support of writing ZIP files to tellable but non-seekable streams. diff --git a/Python/ceval.c b/Python/ceval.c --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1189,8 +1189,8 @@ f->f_stacktop = NULL; /* remains NULL unless yield suspends frame */ f->f_executing = 1; - if (co->co_flags & CO_GENERATOR && !throwflag) { - if (f->f_exc_type != NULL && f->f_exc_type != Py_None) { + if (co->co_flags & CO_GENERATOR) { + if (!throwflag && f->f_exc_type != NULL && f->f_exc_type != Py_None) { /* We were in an except handler when we left, restore the exception state which was put aside (see YIELD_VALUE). */ @@ -3196,7 +3196,8 @@ || (retval == NULL && PyErr_Occurred())); fast_yield: - if (co->co_flags & CO_GENERATOR && (why == WHY_YIELD || why == WHY_RETURN)) { + if (co->co_flags & CO_GENERATOR) { + /* The purpose of this block is to put aside the generator's exception state and restore that of the calling frame. If the current exception state is from the caller, we clear the exception values -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 11:09:06 2015 From: python-checkins at python.org (victor.stinner) Date: Sat, 31 Jan 2015 10:09:06 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzUz?= =?utf-8?q?=3A_Fix_the_exception_handling_of_generators_in_PyEval=5FEvalFr?= =?utf-8?b?YW1lRXgoKS4=?= Message-ID: <20150131100902.96090.88652@psf.io> https://hg.python.org/cpython/rev/4555da8c3091 changeset: 94412:4555da8c3091 branch: 3.4 parent: 94410:5b5a581d91c8 user: Victor Stinner date: Sat Jan 31 10:29:47 2015 +0100 summary: Issue #23353: Fix the exception handling of generators in PyEval_EvalFrameEx(). At entry, save or swap the exception state even if PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception state is now always restored or swapped, not only if why is WHY_YIELD or WHY_RETURN. Patch co-written with Antoine Pitrou. files: Lib/test/test_generators.py | 109 ++++++++++++++++++++++++ Misc/NEWS | 6 + Python/ceval.c | 7 +- 3 files changed, 119 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -50,6 +50,115 @@ self.assertEqual(gc.garbage, old_garbage) +class ExceptionTest(unittest.TestCase): + # Tests for the issue #23353: check that the currently handled exception + # is correctly saved/restored in PyEval_EvalFrameEx(). + + def test_except_throw(self): + def store_raise_exc_generator(): + try: + self.assertEqual(sys.exc_info()[0], None) + yield + except Exception as exc: + # exception raised by gen.throw(exc) + self.assertEqual(sys.exc_info()[0], ValueError) + self.assertIsNone(exc.__context__) + yield + + # ensure that the exception is not lost + self.assertEqual(sys.exc_info()[0], ValueError) + yield + + # we should be able to raise back the ValueError + raise + + make = store_raise_exc_generator() + next(make) + + try: + raise ValueError() + except Exception as exc: + try: + make.throw(exc) + except Exception: + pass + + next(make) + with self.assertRaises(ValueError) as cm: + next(make) + self.assertIsNone(cm.exception.__context__) + + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_next(self): + def gen(): + self.assertEqual(sys.exc_info()[0], ValueError) + yield "done" + + g = gen() + try: + raise ValueError + except Exception: + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_gen_except(self): + def gen(): + try: + self.assertEqual(sys.exc_info()[0], None) + yield + # we are called from "except ValueError:", TypeError must + # inherit ValueError in its context + raise TypeError() + except TypeError as exc: + self.assertEqual(sys.exc_info()[0], TypeError) + self.assertEqual(type(exc.__context__), ValueError) + # here we are still called from the "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + yield + self.assertIsNone(sys.exc_info()[0]) + yield "done" + + g = gen() + next(g) + try: + raise ValueError + except Exception: + next(g) + + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_throw_exception_context(self): + def gen(): + try: + try: + self.assertEqual(sys.exc_info()[0], None) + yield + except ValueError: + # we are called from "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + raise TypeError() + except Exception as exc: + self.assertEqual(sys.exc_info()[0], TypeError) + self.assertEqual(type(exc.__context__), ValueError) + # we are still called from "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + yield + self.assertIsNone(sys.exc_info()[0]) + yield "done" + + g = gen() + next(g) + try: + raise ValueError + except Exception as exc: + g.throw(exc) + + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + tutorial_tests = """ Let's try a simple generator: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -50,6 +50,12 @@ Library ------- +- Issue #23353: Fix the exception handling of generators in + PyEval_EvalFrameEx(). At entry, save or swap the exception state even if + PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception state + is now always restored or swapped, not only if why is WHY_YIELD or + WHY_RETURN. Patch co-written with Antoine Pitrou. + - Issue #18518: timeit now rejects statements which can't be compiled outside a function or a loop (e.g. "return" or "break"). diff --git a/Python/ceval.c b/Python/ceval.c --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1189,8 +1189,8 @@ f->f_stacktop = NULL; /* remains NULL unless yield suspends frame */ f->f_executing = 1; - if (co->co_flags & CO_GENERATOR && !throwflag) { - if (f->f_exc_type != NULL && f->f_exc_type != Py_None) { + if (co->co_flags & CO_GENERATOR) { + if (!throwflag && f->f_exc_type != NULL && f->f_exc_type != Py_None) { /* We were in an except handler when we left, restore the exception state which was put aside (see YIELD_VALUE). */ @@ -3172,7 +3172,8 @@ || (retval == NULL && PyErr_Occurred())); fast_yield: - if (co->co_flags & CO_GENERATOR && (why == WHY_YIELD || why == WHY_RETURN)) { + if (co->co_flags & CO_GENERATOR) { + /* The purpose of this block is to put aside the generator's exception state and restore that of the calling frame. If the current exception state is from the caller, we clear the exception values -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 11:24:05 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 10:24:05 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <20150131102359.96084.76857@psf.io> https://hg.python.org/cpython/rev/2f8ad1424534 changeset: 94415:2f8ad1424534 parent: 94414:3603bae63c13 parent: 94413:cb5cdb0aa18d user: Serhiy Storchaka date: Sat Jan 31 12:23:01 2015 +0200 summary: Merge heads files: Lib/test/test_generators.py | 109 ++++++++++++++++++++++++ Misc/NEWS | 6 + Python/ceval.c | 7 +- 3 files changed, 119 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -89,6 +89,115 @@ "GeneratorTest.test_name..") +class ExceptionTest(unittest.TestCase): + # Tests for the issue #23353: check that the currently handled exception + # is correctly saved/restored in PyEval_EvalFrameEx(). + + def test_except_throw(self): + def store_raise_exc_generator(): + try: + self.assertEqual(sys.exc_info()[0], None) + yield + except Exception as exc: + # exception raised by gen.throw(exc) + self.assertEqual(sys.exc_info()[0], ValueError) + self.assertIsNone(exc.__context__) + yield + + # ensure that the exception is not lost + self.assertEqual(sys.exc_info()[0], ValueError) + yield + + # we should be able to raise back the ValueError + raise + + make = store_raise_exc_generator() + next(make) + + try: + raise ValueError() + except Exception as exc: + try: + make.throw(exc) + except Exception: + pass + + next(make) + with self.assertRaises(ValueError) as cm: + next(make) + self.assertIsNone(cm.exception.__context__) + + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_next(self): + def gen(): + self.assertEqual(sys.exc_info()[0], ValueError) + yield "done" + + g = gen() + try: + raise ValueError + except Exception: + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_gen_except(self): + def gen(): + try: + self.assertEqual(sys.exc_info()[0], None) + yield + # we are called from "except ValueError:", TypeError must + # inherit ValueError in its context + raise TypeError() + except TypeError as exc: + self.assertEqual(sys.exc_info()[0], TypeError) + self.assertEqual(type(exc.__context__), ValueError) + # here we are still called from the "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + yield + self.assertIsNone(sys.exc_info()[0]) + yield "done" + + g = gen() + next(g) + try: + raise ValueError + except Exception: + next(g) + + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + def test_except_throw_exception_context(self): + def gen(): + try: + try: + self.assertEqual(sys.exc_info()[0], None) + yield + except ValueError: + # we are called from "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + raise TypeError() + except Exception as exc: + self.assertEqual(sys.exc_info()[0], TypeError) + self.assertEqual(type(exc.__context__), ValueError) + # we are still called from "except ValueError:" + self.assertEqual(sys.exc_info()[0], ValueError) + yield + self.assertIsNone(sys.exc_info()[0]) + yield "done" + + g = gen() + next(g) + try: + raise ValueError + except Exception as exc: + g.throw(exc) + + self.assertEqual(next(g), "done") + self.assertEqual(sys.exc_info(), (None, None, None)) + + tutorial_tests = """ Let's try a simple generator: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -229,6 +229,12 @@ - Issue #23326: Removed __ne__ implementations. Since fixing default __ne__ implementation in issue #21408 they are redundant. +- Issue #23353: Fix the exception handling of generators in + PyEval_EvalFrameEx(). At entry, save or swap the exception state even if + PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception state + is now always restored or swapped, not only if why is WHY_YIELD or + WHY_RETURN. Patch co-written with Antoine Pitrou. + - Issue #14099: Restored support of writing ZIP files to tellable but non-seekable streams. diff --git a/Python/ceval.c b/Python/ceval.c --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1189,8 +1189,8 @@ f->f_stacktop = NULL; /* remains NULL unless yield suspends frame */ f->f_executing = 1; - if (co->co_flags & CO_GENERATOR && !throwflag) { - if (f->f_exc_type != NULL && f->f_exc_type != Py_None) { + if (co->co_flags & CO_GENERATOR) { + if (!throwflag && f->f_exc_type != NULL && f->f_exc_type != Py_None) { /* We were in an except handler when we left, restore the exception state which was put aside (see YIELD_VALUE). */ @@ -3196,7 +3196,8 @@ || (retval == NULL && PyErr_Occurred())); fast_yield: - if (co->co_flags & CO_GENERATOR && (why == WHY_YIELD || why == WHY_RETURN)) { + if (co->co_flags & CO_GENERATOR) { + /* The purpose of this block is to put aside the generator's exception state and restore that of the calling frame. If the current exception state is from the caller, we clear the exception values -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 11:24:05 2015 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 31 Jan 2015 10:24:05 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323326=3A_Removed_?= =?utf-8?q?=5F=5Fne=5F=5F_implementations=2E__Since_fixing_default_=5F=5Fn?= =?utf-8?b?ZV9f?= Message-ID: <20150131102358.25863.99654@psf.io> https://hg.python.org/cpython/rev/3603bae63c13 changeset: 94414:3603bae63c13 parent: 94411:3813a5282eac user: Serhiy Storchaka date: Sat Jan 31 12:05:05 2015 +0200 summary: Issue #23326: Removed __ne__ implementations. Since fixing default __ne__ implementation in issue #21408 they are redundant. files: Lib/_pydecimal.py | 9 -------- Lib/argparse.py | 5 ---- Lib/asyncio/events.py | 4 --- Lib/collections/__init__.py | 5 ---- Lib/datetime.py | 25 ------------------------ Lib/distutils/version.py | 6 ----- Lib/email/charset.py | 3 -- Lib/email/header.py | 3 -- Lib/functools.py | 2 - Lib/inspect.py | 9 -------- Lib/sched.py | 1 - Lib/test/datetimetester.py | 2 - Lib/test/test_argparse.py | 3 -- Lib/test/test_long.py | 2 - Lib/unittest/mock.py | 4 --- Lib/uuid.py | 5 ---- Lib/xml/dom/minidom.py | 3 -- Lib/xml/etree/ElementTree.py | 4 --- Lib/xmlrpc/client.py | 9 -------- Misc/NEWS | 3 ++ 20 files changed, 3 insertions(+), 104 deletions(-) diff --git a/Lib/_pydecimal.py b/Lib/_pydecimal.py --- a/Lib/_pydecimal.py +++ b/Lib/_pydecimal.py @@ -923,15 +923,6 @@ return False return self._cmp(other) == 0 - def __ne__(self, other, context=None): - self, other = _convert_for_comparison(self, other, equality_op=True) - if other is NotImplemented: - return other - if self._check_nans(other, context): - return True - return self._cmp(other) != 0 - - def __lt__(self, other, context=None): self, other = _convert_for_comparison(self, other) if other is NotImplemented: diff --git a/Lib/argparse.py b/Lib/argparse.py --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -1209,11 +1209,6 @@ return NotImplemented return vars(self) == vars(other) - def __ne__(self, other): - if not isinstance(other, Namespace): - return NotImplemented - return not (self == other) - def __contains__(self, key): return key in self.__dict__ diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -178,10 +178,6 @@ self._cancelled == other._cancelled) return NotImplemented - def __ne__(self, other): - equal = self.__eq__(other) - return NotImplemented if equal is NotImplemented else not equal - def cancel(self): if not self._cancelled: self._loop._timer_handle_cancelled(self) diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -987,7 +987,6 @@ def __lt__(self, other): return self.data < self.__cast(other) def __le__(self, other): return self.data <= self.__cast(other) def __eq__(self, other): return self.data == self.__cast(other) - def __ne__(self, other): return self.data != self.__cast(other) def __gt__(self, other): return self.data > self.__cast(other) def __ge__(self, other): return self.data >= self.__cast(other) def __cast(self, other): @@ -1064,10 +1063,6 @@ if isinstance(string, UserString): return self.data == string.data return self.data == string - def __ne__(self, string): - if isinstance(string, UserString): - return self.data != string.data - return self.data != string def __lt__(self, string): if isinstance(string, UserString): return self.data < string.data diff --git a/Lib/datetime.py b/Lib/datetime.py --- a/Lib/datetime.py +++ b/Lib/datetime.py @@ -567,12 +567,6 @@ else: return False - def __ne__(self, other): - if isinstance(other, timedelta): - return self._cmp(other) != 0 - else: - return True - def __le__(self, other): if isinstance(other, timedelta): return self._cmp(other) <= 0 @@ -804,11 +798,6 @@ return self._cmp(other) == 0 return NotImplemented - def __ne__(self, other): - if isinstance(other, date): - return self._cmp(other) != 0 - return NotImplemented - def __le__(self, other): if isinstance(other, date): return self._cmp(other) <= 0 @@ -1079,12 +1068,6 @@ else: return False - def __ne__(self, other): - if isinstance(other, time): - return self._cmp(other, allow_mixed=True) != 0 - else: - return True - def __le__(self, other): if isinstance(other, time): return self._cmp(other) <= 0 @@ -1651,14 +1634,6 @@ else: return False - def __ne__(self, other): - if isinstance(other, datetime): - return self._cmp(other, allow_mixed=True) != 0 - elif not isinstance(other, date): - return NotImplemented - else: - return True - def __le__(self, other): if isinstance(other, datetime): return self._cmp(other) <= 0 diff --git a/Lib/distutils/version.py b/Lib/distutils/version.py --- a/Lib/distutils/version.py +++ b/Lib/distutils/version.py @@ -48,12 +48,6 @@ return c return c == 0 - def __ne__(self, other): - c = self._cmp(other) - if c is NotImplemented: - return c - return c != 0 - def __lt__(self, other): c = self._cmp(other) if c is NotImplemented: diff --git a/Lib/email/charset.py b/Lib/email/charset.py --- a/Lib/email/charset.py +++ b/Lib/email/charset.py @@ -249,9 +249,6 @@ def __eq__(self, other): return str(self) == str(other).lower() - def __ne__(self, other): - return not self.__eq__(other) - def get_body_encoding(self): """Return the content-transfer-encoding used for body encoding. diff --git a/Lib/email/header.py b/Lib/email/header.py --- a/Lib/email/header.py +++ b/Lib/email/header.py @@ -262,9 +262,6 @@ # args and do another comparison. return other == str(self) - def __ne__(self, other): - return not self == other - def append(self, s, charset=None, errors='strict'): """Append a string to the MIME header. diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -223,8 +223,6 @@ return mycmp(self.obj, other.obj) <= 0 def __ge__(self, other): return mycmp(self.obj, other.obj) >= 0 - def __ne__(self, other): - return mycmp(self.obj, other.obj) != 0 __hash__ = None return K diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -2255,9 +2255,6 @@ self._default == other._default and self._annotation == other._annotation) - def __ne__(self, other): - return not self.__eq__(other) - class BoundArguments: """Result of `Signature.bind` call. Holds the mapping of arguments @@ -2342,9 +2339,6 @@ self.signature == other.signature and self.arguments == other.arguments) - def __ne__(self, other): - return not self.__eq__(other) - class Signature: """A Signature object represents the overall signature of a function. @@ -2559,9 +2553,6 @@ return (isinstance(other, Signature) and self._hash_basis() == other._hash_basis()) - def __ne__(self, other): - return not self.__eq__(other) - def _bind(self, args, kwargs, *, partial=False): """Private method. Don't use directly.""" diff --git a/Lib/sched.py b/Lib/sched.py --- a/Lib/sched.py +++ b/Lib/sched.py @@ -41,7 +41,6 @@ class Event(namedtuple('Event', 'time, priority, action, argument, kwargs')): def __eq__(s, o): return (s.time, s.priority) == (o.time, o.priority) - def __ne__(s, o): return (s.time, s.priority) != (o.time, o.priority) def __lt__(s, o): return (s.time, s.priority) < (o.time, o.priority) def __le__(s, o): return (s.time, s.priority) <= (o.time, o.priority) def __gt__(s, o): return (s.time, s.priority) > (o.time, o.priority) diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -1305,8 +1305,6 @@ return isinstance(other, LargerThanAnything) def __eq__(self, other): return isinstance(other, LargerThanAnything) - def __ne__(self, other): - return not isinstance(other, LargerThanAnything) def __gt__(self, other): return not isinstance(other, LargerThanAnything) def __ge__(self, other): diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -69,9 +69,6 @@ def __eq__(self, other): return vars(self) == vars(other) - def __ne__(self, other): - return not (self == other) - class ArgumentParserError(Exception): diff --git a/Lib/test/test_long.py b/Lib/test/test_long.py --- a/Lib/test/test_long.py +++ b/Lib/test/test_long.py @@ -599,8 +599,6 @@ return (x > y) - (x < y) def __eq__(self, other): return self._cmp__(other) == 0 - def __ne__(self, other): - return self._cmp__(other) != 0 def __ge__(self, other): return self._cmp__(other) >= 0 def __gt__(self, other): diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -2015,10 +2015,6 @@ return (other_args, other_kwargs) == (self_args, self_kwargs) - def __ne__(self, other): - return not self.__eq__(other) - - def __call__(self, *args, **kwargs): if self.name is None: return _Call(('', args, kwargs), name='()') diff --git a/Lib/uuid.py b/Lib/uuid.py --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -185,11 +185,6 @@ return self.int == other.int return NotImplemented - def __ne__(self, other): - if isinstance(other, UUID): - return self.int != other.int - return NotImplemented - # Q. What's the value of being able to sort UUIDs? # A. Use them as keys in a B-Tree or similar mapping. diff --git a/Lib/xml/dom/minidom.py b/Lib/xml/dom/minidom.py --- a/Lib/xml/dom/minidom.py +++ b/Lib/xml/dom/minidom.py @@ -545,9 +545,6 @@ def __lt__(self, other): return self._cmp(other) < 0 - def __ne__(self, other): - return self._cmp(other) != 0 - def __getitem__(self, attname_or_tuple): if isinstance(attname_or_tuple, tuple): return self._attrsNS[attname_or_tuple] diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -532,10 +532,6 @@ if isinstance(other, QName): return self.text == other.text return self.text == other - def __ne__(self, other): - if isinstance(other, QName): - return self.text != other.text - return self.text != other # -------------------------------------------------------------------- diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -340,10 +340,6 @@ s, o = self.make_comparable(other) return s == o - def __ne__(self, other): - s, o = self.make_comparable(other) - return s != o - def timetuple(self): return time.strptime(self.value, "%Y%m%dT%H:%M:%S") @@ -407,11 +403,6 @@ other = other.data return self.data == other - def __ne__(self, other): - if isinstance(other, Binary): - other = other.data - return self.data != other - def decode(self, data): self.data = base64.decodebytes(data) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -226,6 +226,9 @@ Library ------- +- Issue #23326: Removed __ne__ implementations. Since fixing default __ne__ + implementation in issue #21408 they are redundant. + - Issue #14099: Restored support of writing ZIP files to tellable but non-seekable streams. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 11:45:31 2015 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 31 Jan 2015 10:45:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Keep_the_definition_of_i_c?= =?utf-8?q?onsistent_between_set=5Flookkey=28=29_and?= Message-ID: <20150131104530.106370.30383@psf.io> https://hg.python.org/cpython/rev/33db20c8537c changeset: 94416:33db20c8537c user: Raymond Hettinger date: Sat Jan 31 02:45:12 2015 -0800 summary: Keep the definition of i consistent between set_lookkey() and set_insert_clean(). files: Objects/setobject.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -56,11 +56,11 @@ setentry *entry; size_t perturb = hash; size_t mask = so->mask; - size_t i = (size_t)hash; /* Unsigned for defined overflow behavior. */ + size_t i = (size_t)hash & mask; /* Unsigned for defined overflow behavior */ size_t j; int cmp; - entry = &table[i & mask]; + entry = &table[i]; if (entry->key == NULL) return entry; @@ -116,9 +116,9 @@ } perturb >>= PERTURB_SHIFT; - i = i * 5 + 1 + perturb; + i = (i * 5 + 1 + perturb) & mask; - entry = &table[i & mask]; + entry = &table[i]; if (entry->key == NULL) goto found_null; } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 20:20:41 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 31 Jan 2015 19:20:41 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fixes_python3=5Fd=2Edll_bu?= =?utf-8?q?ild_so_that_functions_are_forwarded_to_python35=5Fd=2Edll?= Message-ID: <20150131192038.25841.38015@psf.io> https://hg.python.org/cpython/rev/ce5fce6b71b1 changeset: 94417:ce5fce6b71b1 user: Steve Dower date: Sat Jan 31 11:17:07 2015 -0800 summary: Fixes python3_d.dll build so that functions are forwarded to python35_d.dll files: PCbuild/python3dll.vcxproj | 35 +++++++++++++++++++++++-- 1 files changed, 32 insertions(+), 3 deletions(-) diff --git a/PCbuild/python3dll.vcxproj b/PCbuild/python3dll.vcxproj --- a/PCbuild/python3dll.vcxproj +++ b/PCbuild/python3dll.vcxproj @@ -66,7 +66,8 @@ $(OutDir)$(TargetName)stub.lib - $(PySourcePath)PC\python3.def + $(PySourcePath)PC\python3.def + $(IntDir)python3_d.def DllMain @@ -87,9 +88,37 @@ + + + + <_DefLines Remove="@(_DefLines)" /> + <_Lines Remove="@(_Lines)" /> + + + + + + <_Pattern1>(=python$(MajorVersionNumber)$(MinorVersionNumber))\. + <_Sub1>$1_d. + <_Pattern2>"python3" + <_Sub2>"python3_d" + + + <_Lines Include="@(_DefLines)"> + $([System.Text.RegularExpressions.Regex]::Replace($([System.Text.RegularExpressions.Regex]::Replace(`%(Identity)`, `$(_Pattern1)`, `$(_Sub1)`)), `$(_Pattern2)`, `$(_Sub2)`)) + + + + + + + + <_DefLines Remove="@(_DefLines)" /> + <_Lines Remove="@(_Lines)" /> + - + <_Pattern>^[\w.]+=.+?\.([^ ]+).*$ @@ -97,7 +126,7 @@ <_Lines Include="EXPORTS" /> - <_Symbols Include="@(DefLines)" Condition="$([System.Text.RegularExpressions.Regex]::IsMatch(`%(Identity)`, `$(_Pattern)`))"> + <_Symbols Include="@(_DefLines)" Condition="$([System.Text.RegularExpressions.Regex]::IsMatch(`%(Identity)`, `$(_Pattern)`))"> $([System.Text.RegularExpressions.Regex]::Replace(`%(Identity)`, `$(_Pattern)`, `$(_Sub)`)) <_Lines Include="@(_Symbols->'%(Symbol)')" /> -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 21:28:26 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 31 Jan 2015 20:28:26 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMTk5?= =?utf-8?q?=3A_libpython27=2Ea_in_amd64_release_is_32-bit?= Message-ID: <20150131202826.106293.38483@psf.io> https://hg.python.org/cpython/rev/536a816e7232 changeset: 94418:536a816e7232 branch: 2.7 parent: 94409:47a9cb7ec0cb user: Steve Dower date: Sat Jan 31 12:15:13 2015 -0800 summary: Issue #23199: libpython27.a in amd64 release is 32-bit files: Tools/msi/msi.py | 37 +++++++++++++++++------------------ 1 files changed, 18 insertions(+), 19 deletions(-) diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -132,34 +132,32 @@ # Build the mingw import library, libpythonXY.a # This requires 'nm' and 'dlltool' executables on your PATH -def build_mingw_lib(lib_file, def_file, dll_file, mingw_lib): +def build_mingw_lib(dll_path, def_file, dll_file, mingw_lib): warning = "WARNING: %s - libpythonXX.a not built" - nm = find_executable('nm') + gendef = find_executable('gendef') dlltool = find_executable('dlltool') - if not nm or not dlltool: - print warning % "nm and/or dlltool were not found" + if not gendef or not dlltool: + print warning % "gendef and/or dlltool were not found" return False - nm_command = '%s -Cs %s' % (nm, lib_file) + gendef_command = '%s - %s' % (gendef, dll_path) dlltool_command = "%s --dllname %s --def %s --output-lib %s" % \ (dlltool, dll_file, def_file, mingw_lib) - export_match = re.compile(r"^_imp__(.*) in python\d+\.dll").match + if msilib.Win64: + dlltool_command += " -m i386:x86-64" + else: + dlltool_command += " -m i386" f = open(def_file,'w') - print >>f, "LIBRARY %s" % dll_file - print >>f, "EXPORTS" - - nm_pipe = os.popen(nm_command) - for line in nm_pipe.readlines(): - m = export_match(line) - if m: - print >>f, m.group(1) + gendef_pipe = os.popen(gendef_command) + for line in gendef_pipe.readlines(): + print >>f, line f.close() - exit = nm_pipe.close() + exit = gendef_pipe.close() if exit: - print warning % "nm did not run successfully" + print warning % "gendef did not run successfully" return False if os.system(dlltool_command) != 0: @@ -169,13 +167,11 @@ return True # Target files (.def and .a) go in PCBuild directory -lib_file = os.path.join(srcdir, PCBUILD, "python%s%s.lib" % (major, minor)) +dll_path = os.path.join(srcdir, PCBUILD, "python%s%s.dll" % (major, minor)) def_file = os.path.join(srcdir, PCBUILD, "python%s%s.def" % (major, minor)) dll_file = "python%s%s.dll" % (major, minor) mingw_lib = os.path.join(srcdir, PCBUILD, "libpython%s%s.a" % (major, minor)) -have_mingw = build_mingw_lib(lib_file, def_file, dll_file, mingw_lib) - # Determine the target architecture if os.system("nmake /nologo /c /f msisupport.mak") != 0: raise RuntimeError("'nmake /f msisupport.mak' failed") @@ -183,6 +179,7 @@ msilib.set_arch_from_file(dll_path) if msilib.pe_type(dll_path) != msilib.pe_type("msisupport.dll"): raise SystemError, "msisupport.dll for incorrect architecture" + if msilib.Win64: upgrade_code = upgrade_code_64 # Bump the last digit of the code by one, so that 32-bit and 64-bit @@ -190,6 +187,8 @@ digit = hex((int(product_code[-2],16)+1)%16)[-1] product_code = product_code[:-2] + digit + '}' +have_mingw = build_mingw_lib(dll_path, def_file, dll_file, mingw_lib) + if testpackage: ext = 'px' testprefix = 'x' -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 21:29:00 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 31 Jan 2015 20:29:00 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMjEy?= =?utf-8?q?=3A_Update_Windows_copy_of_OpenSSL_to_1=2E0=2E1l?= Message-ID: <20150131202853.25849.40362@psf.io> https://hg.python.org/cpython/rev/c8d7df3cb854 changeset: 94419:c8d7df3cb854 branch: 2.7 user: Steve Dower date: Sat Jan 31 12:18:33 2015 -0800 summary: Issue #23212: Update Windows copy of OpenSSL to 1.0.1l files: PCbuild/pyproject.vsprops | 2 +- Tools/buildbot/external-common.bat | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/PCbuild/pyproject.vsprops b/PCbuild/pyproject.vsprops --- a/PCbuild/pyproject.vsprops +++ b/PCbuild/pyproject.vsprops @@ -82,7 +82,7 @@ /> https://hg.python.org/cpython/rev/8049954bc0e4 changeset: 94420:8049954bc0e4 branch: 3.4 parent: 94412:4555da8c3091 user: Steve Dower date: Sat Jan 31 12:19:10 2015 -0800 summary: Issue #23212: Update Windows copy of OpenSSL to 1.0.1l files: PCbuild/pyproject.props | 2 +- Tools/buildbot/external-common.bat | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props --- a/PCbuild/pyproject.props +++ b/PCbuild/pyproject.props @@ -20,7 +20,7 @@ $(externalsDir)\sqlite-3.8.3.1 $(externalsDir)\bzip2-1.0.6 $(externalsDir)\xz-5.0.5 - $(externalsDir)\openssl-1.0.1j + $(externalsDir)\openssl-1.0.1l $(externalsDir)\tcltk $(externalsDir)\tcltk64 $(tcltkDir)\lib\tcl86t.lib;$(tcltkDir)\lib\tk86t.lib diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -14,7 +14,7 @@ @rem if exist tk-8.6.1.0 rd /s/q tk-8.6.1.0 @rem if exist tix-8.4.3.4 rd /s/q tix-8.4.3.4 @rem if exist db-4.4.20 rd /s/q db-4.4.20 - at rem if exist openssl-1.0.1j rd /s/q openssl-1.0.1j + at rem if exist openssl-1.0.1l rd /s/q openssl-1.0.1l @rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12 @rem bzip @@ -28,9 +28,9 @@ if not exist nasm-2.11.06 svn export %SVNROOT%nasm-2.11.06 @rem OpenSSL -if not exist openssl-1.0.1j ( - rd /s/q openssl-1.0.1i - svn export %SVNROOT%openssl-1.0.1j +if not exist openssl-1.0.1l ( + rd /s/q openssl-1.0.1j + svn export %SVNROOT%openssl-1.0.1l ) @rem tcl/tk -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 21:31:44 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 31 Jan 2015 20:31:44 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_with_3=2E4?= Message-ID: <20150131203144.34382.86158@psf.io> https://hg.python.org/cpython/rev/a3a1e44f2253 changeset: 94421:a3a1e44f2253 parent: 94417:ce5fce6b71b1 parent: 94420:8049954bc0e4 user: Steve Dower date: Sat Jan 31 12:30:22 2015 -0800 summary: Null merge with 3.4 files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Jan 31 21:31:45 2015 From: python-checkins at python.org (steve.dower) Date: Sat, 31 Jan 2015 20:31:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323212=3A_Update_W?= =?utf-8?q?indows_copy_of_OpenSSL_to_1=2E0=2E1l?= Message-ID: <20150131203145.39292.10628@psf.io> https://hg.python.org/cpython/rev/469ff344f8fd changeset: 94422:469ff344f8fd user: Steve Dower date: Sat Jan 31 12:20:40 2015 -0800 summary: Issue #23212: Update Windows copy of OpenSSL to 1.0.1l files: PCbuild/get_externals.bat | 2 +- PCbuild/python.props | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -54,7 +54,7 @@ for %%e in ( bzip2-1.0.6 nasm-2.11.06 - openssl-1.0.1j + openssl-1.0.1l tcl-core-8.6.3.1 tk-8.6.3.1 tix-8.4.3.4 diff --git a/PCbuild/python.props b/PCbuild/python.props --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -34,7 +34,7 @@ $(ExternalsDir)sqlite-3.8.3.1\ $(ExternalsDir)bzip2-1.0.6\ $(ExternalsDir)xz-5.0.5\ - $(ExternalsDir)openssl-1.0.1j\ + $(ExternalsDir)openssl-1.0.1l\ $(ExternalsDir)\nasm-2.11.06\ -- Repository URL: https://hg.python.org/cpython