From python-checkins at python.org Mon Dec 1 00:17:45 2014 From: python-checkins at python.org (donald.stufft) Date: Sun, 30 Nov 2014 23:17:45 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Spelling_fixes?= Message-ID: <20141130231745.84305.34016@psf.io> https://hg.python.org/peps/rev/1162102a0229 changeset: 5632:1162102a0229 user: Donald Stufft date: Sun Nov 30 18:17:40 2014 -0500 summary: Spelling fixes files: pep-0481.txt | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-) diff --git a/pep-0481.txt b/pep-0481.txt --- a/pep-0481.txt +++ b/pep-0481.txt @@ -50,7 +50,7 @@ them to peps at python.org. This process is unfriendly towards non-comitter contributors as well as cumbersome for comitters seeking to accept the patches sent by users. In contrast, the Pull Request workflow style enables non -techincal contributors, especially those who do not know their way around the +technical contributors, especially those who do not know their way around the DVCS of choice, to contribute using the web based editor. On the committer side, the Pull Requests enable them to tell, before merging, whether or not a particular Pull Request will break anything. It also enables them to do a @@ -66,7 +66,7 @@ -------- Looking at the variety of DVCS which are available today, it becomes fairly -clear that git has the largest mindshare. The Open Hub (Previously Ohloh) +clear that git has the largest mindshare. The Open Hub (previously Ohloh) statistics [#openhub-stats]_ show that currently 37% of the repositories indexed by Open Hub are using git which is second only to SVN (which has 48%), while Mercurial has just 2% of the indexed repositories (beating only bazaar @@ -88,8 +88,8 @@ In addition to simply making it more likely that any individual will already know how to use git, the number of projects and people using it means that the resources for learning the tool are likely to be more fully fleshed out. -When you run into problems, the liklihood that someone else had that problem -and posted a question and recieved an answer is also far likelier. +When you run into problems, the likelihood that someone else had that problem +and posted a question and recieved an answer is also far higher. Thirdly, by using a more popular tool you also increase your options for tooling *around* the DVCS itself. Looking at the various options for hosting @@ -99,7 +99,7 @@ Therefore the popularity of git increases the flexibility of our options going into the future for what toolchain these projects use. -Also, by moving to the more popular DVCS, we increase the likelhood that the +Also, by moving to the more popular DVCS, we increase the likelihood that the knowledge that the person has learned in contributing to these support repositories will transfer to projects outside of the immediate CPython project such as to the larger Python community which is primarily using Git hosted on @@ -108,7 +108,7 @@ In previous years there was concern about how well supported git was on Windows in comparison to Mercurial. However, git has grown to support Windows as a first class citizen. In addition to that, for Windows users who are not well -aquanted with the Windows command line, there are GUI options as well. +acquainted with the Windows command line, there are GUI options as well. Why Github? @@ -116,7 +116,7 @@ There are a number of software projects or web services which offer functionality similar to that of Github. These range from commerical web -services such as a Bitbucket to self-hosted OSS solutions such as Kallithea or +services such as Bitbucket to self-hosted OSS solutions such as Kallithea or Gitlab. This PEP proposes that we move these repositories to Github. There are two primary reasons for selecting Github: Popularity and @@ -124,7 +124,7 @@ Github is currently the most popular hosted repository hosting according to Alexa, where it currently has a global rank of 121. Much like for Git itself, -by choosing the most popular tool we gain benefits in increasing the likelhood +by choosing the most popular tool we gain benefits in increasing the likelihood that a new contributor will have already experienced the toolchain, the quality and availablity of the help, more and better tooling being built around it, and the knowledge transfer to other projects. A look again at the top 100 projects @@ -145,10 +145,10 @@ Finally, a reason to choose a web service at all over something that is self-hosted is to be able to more efficiently use volunteer time and donated -resources. Every additional service hosted on the PSF infrastruture by the +resources. Every additional service hosted on the PSF infrastructure by the PSF infrastructure team further spreads out the amount of time that the volunteers on that team have to spend and uses some chunk of resources that -could potentionally be used for something where there is no free or affordable +could potentially be used for something where there is no free or affordable hosted solution available. One concern that people do have with using a hosted service is that there is a @@ -178,7 +178,7 @@ sub-community without having to learn a special, bespoke workflow and a different toolchain for each project. They've found that when people can use their limited time on actually contributing instead of learning the different -tools and workflows that, not only do they contribute more to one project, but +tools and workflows, not only do they contribute more to one project, but that they also expand out and contribute to other projects. This move has also been attributed to the increased tendency for members of that community to go so far as publishing their research and educational materials on Github as @@ -195,7 +195,7 @@ Through the use of hg-git [#hg-git]_ we can easily convert a Mercurial repository to a Git repository by simply pushing the Mercurial repository to -the Git repository. People who wish to continue to use Mercurual locally can +the Git repository. People who wish to continue to use Mercurial locally can then use hg-git going into the future using the new Github URL. However they will need to re-clone their repositories as using Git as the server seems to trigger a one time change of the changeset ids. @@ -210,8 +210,8 @@ In addition to the migration of the repository hosting itself there are a number of locations for each particular repository which will require updating. -The bulk of these will simply be changing commands from the hg equivilant to -the git equivilant. +The bulk of these will simply be changing commands from the hg equivalent to +the git equivalent. In particular this will include: -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Mon Dec 1 09:46:09 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 08:46:09 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIyOTQz?= =?utf-8?q?=3A_bsddb_tests_are_locale_independend_now=2E?= Message-ID: <20141201084608.84279.73865@psf.io> https://hg.python.org/cpython/rev/df17d2b0878f changeset: 93670:df17d2b0878f branch: 2.7 parent: 93668:961145c548e2 user: Serhiy Storchaka date: Mon Dec 01 10:34:23 2014 +0200 summary: Issue #22943: bsddb tests are locale independend now. This fixes tests on 8-bit locales (in particular on Windows). files: Lib/bsddb/test/test_all.py | 3 --- Lib/bsddb/test/test_basics.py | 4 ++-- Lib/bsddb/test/test_dbshelve.py | 2 +- Lib/bsddb/test/test_get_none.py | 16 ++++++++-------- Lib/bsddb/test/test_queue.py | 13 ++++++------- Lib/bsddb/test/test_recno.py | 7 +++---- Misc/NEWS | 5 +++++ 7 files changed, 25 insertions(+), 25 deletions(-) diff --git a/Lib/bsddb/test/test_all.py b/Lib/bsddb/test/test_all.py --- a/Lib/bsddb/test/test_all.py +++ b/Lib/bsddb/test/test_all.py @@ -412,9 +412,6 @@ def get_dbp(self) : return self._db - import string - string.letters=[chr(i) for i in xrange(65,91)] - bsddb._db.DBEnv_orig = bsddb._db.DBEnv bsddb._db.DB_orig = bsddb._db.DB if bsddb.db.version() <= (4, 3) : diff --git a/Lib/bsddb/test/test_basics.py b/Lib/bsddb/test/test_basics.py --- a/Lib/bsddb/test/test_basics.py +++ b/Lib/bsddb/test/test_basics.py @@ -999,7 +999,7 @@ for x in "The quick brown fox jumped over the lazy dog".split(): d2.put(x, self.makeData(x)) - for x in string.letters: + for x in string.ascii_letters: d3.put(x, x*70) d1.sync() @@ -1047,7 +1047,7 @@ if verbose: print rec rec = c3.next() - self.assertEqual(count, len(string.letters)) + self.assertEqual(count, len(string.ascii_letters)) c1.close() diff --git a/Lib/bsddb/test/test_dbshelve.py b/Lib/bsddb/test/test_dbshelve.py --- a/Lib/bsddb/test/test_dbshelve.py +++ b/Lib/bsddb/test/test_dbshelve.py @@ -59,7 +59,7 @@ return bytes(key, "iso8859-1") # 8 bits def populateDB(self, d): - for x in string.letters: + for x in string.ascii_letters: d[self.mk('S' + x)] = 10 * x # add a string d[self.mk('I' + x)] = ord(x) # add an integer d[self.mk('L' + x)] = [x] * 10 # add a list diff --git a/Lib/bsddb/test/test_get_none.py b/Lib/bsddb/test/test_get_none.py --- a/Lib/bsddb/test/test_get_none.py +++ b/Lib/bsddb/test/test_get_none.py @@ -26,14 +26,14 @@ d.open(self.filename, db.DB_BTREE, db.DB_CREATE) d.set_get_returns_none(1) - for x in string.letters: + for x in string.ascii_letters: d.put(x, x * 40) data = d.get('bad key') self.assertEqual(data, None) - data = d.get(string.letters[0]) - self.assertEqual(data, string.letters[0]*40) + data = d.get(string.ascii_letters[0]) + self.assertEqual(data, string.ascii_letters[0]*40) count = 0 c = d.cursor() @@ -43,7 +43,7 @@ rec = c.next() self.assertEqual(rec, None) - self.assertEqual(count, len(string.letters)) + self.assertEqual(count, len(string.ascii_letters)) c.close() d.close() @@ -54,14 +54,14 @@ d.open(self.filename, db.DB_BTREE, db.DB_CREATE) d.set_get_returns_none(0) - for x in string.letters: + for x in string.ascii_letters: d.put(x, x * 40) self.assertRaises(db.DBNotFoundError, d.get, 'bad key') self.assertRaises(KeyError, d.get, 'bad key') - data = d.get(string.letters[0]) - self.assertEqual(data, string.letters[0]*40) + data = d.get(string.ascii_letters[0]) + self.assertEqual(data, string.ascii_letters[0]*40) count = 0 exceptionHappened = 0 @@ -77,7 +77,7 @@ self.assertNotEqual(rec, None) self.assertTrue(exceptionHappened) - self.assertEqual(count, len(string.letters)) + self.assertEqual(count, len(string.ascii_letters)) c.close() d.close() diff --git a/Lib/bsddb/test/test_queue.py b/Lib/bsddb/test/test_queue.py --- a/Lib/bsddb/test/test_queue.py +++ b/Lib/bsddb/test/test_queue.py @@ -10,7 +10,6 @@ #---------------------------------------------------------------------- - at unittest.skip("fails on Windows; see issue 22943") class SimpleQueueTestCase(unittest.TestCase): def setUp(self): self.filename = get_new_database_path() @@ -37,17 +36,17 @@ print "before appends" + '-' * 30 pprint(d.stat()) - for x in string.letters: + for x in string.ascii_letters: d.append(x * 40) - self.assertEqual(len(d), len(string.letters)) + self.assertEqual(len(d), len(string.ascii_letters)) d.put(100, "some more data") d.put(101, "and some more ") d.put(75, "out of order") d.put(1, "replacement data") - self.assertEqual(len(d), len(string.letters)+3) + self.assertEqual(len(d), len(string.ascii_letters)+3) if verbose: print "before close" + '-' * 30 @@ -108,17 +107,17 @@ print "before appends" + '-' * 30 pprint(d.stat()) - for x in string.letters: + for x in string.ascii_letters: d.append(x * 40) - self.assertEqual(len(d), len(string.letters)) + self.assertEqual(len(d), len(string.ascii_letters)) d.put(100, "some more data") d.put(101, "and some more ") d.put(75, "out of order") d.put(1, "replacement data") - self.assertEqual(len(d), len(string.letters)+3) + self.assertEqual(len(d), len(string.ascii_letters)+3) if verbose: print "before close" + '-' * 30 diff --git a/Lib/bsddb/test/test_recno.py b/Lib/bsddb/test/test_recno.py --- a/Lib/bsddb/test/test_recno.py +++ b/Lib/bsddb/test/test_recno.py @@ -4,12 +4,11 @@ import os, sys import errno from pprint import pprint +import string import unittest from test_all import db, test_support, verbose, get_new_environment_path, get_new_database_path -letters = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' - #---------------------------------------------------------------------- @@ -39,7 +38,7 @@ d.open(self.filename, db.DB_RECNO, db.DB_CREATE) - for x in letters: + for x in string.ascii_letters: recno = d.append(x * 60) self.assertIsInstance(recno, int) self.assertGreaterEqual(recno, 1) @@ -270,7 +269,7 @@ d.set_re_pad(45) # ...test both int and char d.open(self.filename, db.DB_RECNO, db.DB_CREATE) - for x in letters: + for x in string.ascii_letters: d.append(x * 35) # These will be padded d.append('.' * 40) # this one will be exact diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -30,6 +30,11 @@ - Issue #18905: "pydoc -p 0" now outputs actually used port. Based on patch by Wieland Hoffmann. +Tests +----- + +- Issue #22943: bsddb tests are locale independend now. + What's New in Python 2.7.9? =========================== -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 09:54:00 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 08:54:00 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyOTI0?= =?utf-8?q?=3A_Scripts_gprof2html=2Epy_and_highlight=2Epy_now_use_html=2Ee?= =?utf-8?q?scape=28=29?= Message-ID: <20141201085359.69779.65975@psf.io> https://hg.python.org/cpython/rev/f5eb62bdcb1a changeset: 93671:f5eb62bdcb1a branch: 3.4 parent: 93665:d1f7c3f45ffe user: Serhiy Storchaka date: Mon Dec 01 10:50:33 2014 +0200 summary: Issue #22924: Scripts gprof2html.py and highlight.py now use html.escape() instead of deperecated cgi.escape(). Original patch by Raymond Hettinger. files: Tools/scripts/gprof2html.py | 8 ++++++-- Tools/scripts/highlight.py | 21 +++++++++++++-------- 2 files changed, 19 insertions(+), 10 deletions(-) diff --git a/Tools/scripts/gprof2html.py b/Tools/scripts/gprof2html.py --- a/Tools/scripts/gprof2html.py +++ b/Tools/scripts/gprof2html.py @@ -2,7 +2,11 @@ """Transform gprof(1) output into useful HTML.""" -import re, os, sys, cgi, webbrowser +import html +import os +import re +import sys +import webbrowser header = """\ @@ -22,7 +26,7 @@ def add_escapes(filename): with open(filename) as fp: for line in fp: - yield cgi.escape(line) + yield html.escape(line) def main(): diff --git a/Tools/scripts/highlight.py b/Tools/scripts/highlight.py --- a/Tools/scripts/highlight.py +++ b/Tools/scripts/highlight.py @@ -3,11 +3,12 @@ __author__ = 'Raymond Hettinger' -import keyword, tokenize, cgi, re, functools -try: - import builtins -except ImportError: - import __builtin__ as builtins +import builtins +import functools +import html as html_module +import keyword +import re +import tokenize #### Analyze Python Source ################################# @@ -101,7 +102,7 @@ for kind, text in classified_text: if kind: result.append('' % kind) - result.append(cgi.escape(text)) + result.append(html_module.escape(text)) if kind: result.append('') result.append(closer) @@ -140,7 +141,7 @@ 'Create a complete HTML page with colorized source code' css_str = '\n'.join(['%s %s' % item for item in css.items()]) result = html_highlight(classified_text) - title = cgi.escape(title) + title = html_module.escape(title) return html.format(title=title, css=css_str, body=result) #### LaTeX Output ########################################## @@ -193,7 +194,11 @@ if __name__ == '__main__': - import sys, argparse, webbrowser, os, textwrap + import argparse + import os.path + import sys + import textwrap + import webbrowser parser = argparse.ArgumentParser( description = 'Add syntax highlighting to Python source code', -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 09:54:01 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 08:54:01 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322924=3A_Scripts_gprof2html=2Epy_and_highlight?= =?utf-8?q?=2Epy_now_use_html=2Eescape=28=29?= Message-ID: <20141201085400.55129.51575@psf.io> https://hg.python.org/cpython/rev/8c1d1e861081 changeset: 93672:8c1d1e861081 parent: 93669:64bb01bce12c parent: 93671:f5eb62bdcb1a user: Serhiy Storchaka date: Mon Dec 01 10:51:37 2014 +0200 summary: Issue #22924: Scripts gprof2html.py and highlight.py now use html.escape() instead of deperecated cgi.escape(). Original patch by Raymond Hettinger. files: Tools/scripts/gprof2html.py | 8 ++++++-- Tools/scripts/highlight.py | 21 +++++++++++++-------- 2 files changed, 19 insertions(+), 10 deletions(-) diff --git a/Tools/scripts/gprof2html.py b/Tools/scripts/gprof2html.py --- a/Tools/scripts/gprof2html.py +++ b/Tools/scripts/gprof2html.py @@ -2,7 +2,11 @@ """Transform gprof(1) output into useful HTML.""" -import re, os, sys, cgi, webbrowser +import html +import os +import re +import sys +import webbrowser header = """\ @@ -22,7 +26,7 @@ def add_escapes(filename): with open(filename) as fp: for line in fp: - yield cgi.escape(line) + yield html.escape(line) def main(): diff --git a/Tools/scripts/highlight.py b/Tools/scripts/highlight.py --- a/Tools/scripts/highlight.py +++ b/Tools/scripts/highlight.py @@ -3,11 +3,12 @@ __author__ = 'Raymond Hettinger' -import keyword, tokenize, cgi, re, functools -try: - import builtins -except ImportError: - import __builtin__ as builtins +import builtins +import functools +import html as html_module +import keyword +import re +import tokenize #### Analyze Python Source ################################# @@ -101,7 +102,7 @@ for kind, text in classified_text: if kind: result.append('' % kind) - result.append(cgi.escape(text)) + result.append(html_module.escape(text)) if kind: result.append('') result.append(closer) @@ -140,7 +141,7 @@ 'Create a complete HTML page with colorized source code' css_str = '\n'.join(['%s %s' % item for item in css.items()]) result = html_highlight(classified_text) - title = cgi.escape(title) + title = html_module.escape(title) return html.format(title=title, css=css_str, body=result) #### LaTeX Output ########################################## @@ -193,7 +194,11 @@ if __name__ == '__main__': - import sys, argparse, webbrowser, os, textwrap + import argparse + import os.path + import sys + import textwrap + import webbrowser parser = argparse.ArgumentParser( description = 'Add syntax highlighting to Python source code', -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 10:10:43 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 09:10:43 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322838=3A_All_test=5Fre_tests_now_work_with_unit?= =?utf-8?q?test_test_discovery=2E?= Message-ID: <20141201091041.69795.65838@psf.io> https://hg.python.org/cpython/rev/258ee94b5bac changeset: 93674:258ee94b5bac parent: 93672:8c1d1e861081 parent: 93673:21a7a92f4d0c user: Serhiy Storchaka date: Mon Dec 01 11:08:27 2014 +0200 summary: Issue #22838: All test_re tests now work with unittest test discovery. files: Lib/test/re_tests.py | 2 +- Lib/test/test_re.py | 142 +++++++++++++----------------- Misc/NEWS | 2 + 3 files changed, 64 insertions(+), 82 deletions(-) diff --git a/Lib/test/re_tests.py b/Lib/test/re_tests.py --- a/Lib/test/re_tests.py +++ b/Lib/test/re_tests.py @@ -86,7 +86,7 @@ (r'\a[\b]\f\n\r\t\v', '\a\b\f\n\r\t\v', SUCCEED, 'found', '\a\b\f\n\r\t\v'), (r'[\a][\b][\f][\n][\r][\t][\v]', '\a\b\f\n\r\t\v', SUCCEED, 'found', '\a\b\f\n\r\t\v'), # NOTE: not an error under PCRE/PRE: - # (r'\u', '', SYNTAX_ERROR), # A Perl escape + (r'\u', '', SYNTAX_ERROR), # A Perl escape (r'\c\e\g\h\i\j\k\m\o\p\q\y\z', 'ceghijkmopqyz', SUCCEED, 'found', 'ceghijkmopqyz'), (r'\xff', '\377', SUCCEED, 'found', chr(255)), # new \x semantics diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1511,55 +1511,55 @@ self.assertEqual(f("abcabdac"), [0, 0, 0, 1, 2, 0, 1, 0]) -def run_re_tests(): - from test.re_tests import tests, SUCCEED, FAIL, SYNTAX_ERROR - if verbose: - print('Running re_tests test suite') - else: - # To save time, only run the first and last 10 tests - #tests = tests[:10] + tests[-10:] - pass +class ExternalTests(unittest.TestCase): - for t in tests: - sys.stdout.flush() - pattern = s = outcome = repl = expected = None - if len(t) == 5: - pattern, s, outcome, repl, expected = t - elif len(t) == 3: - pattern, s, outcome = t - else: - raise ValueError('Test tuples should have 3 or 5 fields', t) + def test_re_benchmarks(self): + 're_tests benchmarks' + from test.re_tests import benchmarks + for pattern, s in benchmarks: + with self.subTest(pattern=pattern, string=s): + p = re.compile(pattern) + self.assertTrue(p.search(s)) + self.assertTrue(p.match(s)) + self.assertTrue(p.fullmatch(s)) + s2 = ' '*10000 + s + ' '*10000 + self.assertTrue(p.search(s2)) + self.assertTrue(p.match(s2, 10000)) + self.assertTrue(p.match(s2, 10000, 10000 + len(s))) + self.assertTrue(p.fullmatch(s2, 10000, 10000 + len(s))) - try: - obj = re.compile(pattern) - except re.error: - if outcome == SYNTAX_ERROR: pass # Expected a syntax error + def test_re_tests(self): + 're_tests test suite' + from test.re_tests import tests, SUCCEED, FAIL, SYNTAX_ERROR + for t in tests: + pattern = s = outcome = repl = expected = None + if len(t) == 5: + pattern, s, outcome, repl, expected = t + elif len(t) == 3: + pattern, s, outcome = t else: - print('=== Syntax error:', t) - except KeyboardInterrupt: raise KeyboardInterrupt - except: - print('*** Unexpected error ***', t) - if verbose: - traceback.print_exc(file=sys.stdout) - else: - try: + raise ValueError('Test tuples should have 3 or 5 fields', t) + + with self.subTest(pattern=pattern, string=s): + if outcome == SYNTAX_ERROR: # Expected a syntax error + with self.assertRaises(re.error): + re.compile(pattern) + continue + + obj = re.compile(pattern) result = obj.search(s) - except re.error as msg: - print('=== Unexpected exception', t, repr(msg)) - if outcome == SYNTAX_ERROR: - # This should have been a syntax error; forget it. - pass - elif outcome == FAIL: - if result is None: pass # No match, as expected - else: print('=== Succeeded incorrectly', t) - elif outcome == SUCCEED: - if result is not None: + if outcome == FAIL: + self.assertIsNone(result, 'Succeeded incorrectly') + continue + + with self.subTest(): + self.assertTrue(result, 'Failed incorrectly') # Matched, as expected, so now we compute the # result string and compare it to our expected result. start, end = result.span(0) - vardict={'found': result.group(0), - 'groups': result.group(), - 'flags': result.re.flags} + vardict = {'found': result.group(0), + 'groups': result.group(), + 'flags': result.re.flags} for i in range(1, 100): try: gi = result.group(i) @@ -1577,12 +1577,8 @@ except IndexError: gi = "Error" vardict[i] = gi - repl = eval(repl, vardict) - if repl != expected: - print('=== grouping error', t, end=' ') - print(repr(repl) + ' should be ' + repr(expected)) - else: - print('=== Failed incorrectly', t) + self.assertEqual(eval(repl, vardict), expected, + 'grouping error') # Try the match with both pattern and string converted to # bytes, and check that it still succeeds. @@ -1593,55 +1589,39 @@ # skip non-ascii tests pass else: - try: + with self.subTest('bytes pattern match'): bpat = re.compile(bpat) - except Exception: - print('=== Fails on bytes pattern compile', t) - if verbose: - traceback.print_exc(file=sys.stdout) - else: - bytes_result = bpat.search(bs) - if bytes_result is None: - print('=== Fails on bytes pattern match', t) + self.assertTrue(bpat.search(bs)) # Try the match with the search area limited to the extent # of the match and see if it still succeeds. \B will # break (because it won't match at the end or start of a # string), so we'll ignore patterns that feature it. - - if pattern[:2] != '\\B' and pattern[-2:] != '\\B' \ - and result is not None: - obj = re.compile(pattern) - result = obj.search(s, result.start(0), result.end(0) + 1) - if result is None: - print('=== Failed on range-limited match', t) + if (pattern[:2] != r'\B' and pattern[-2:] != r'\B' + and result is not None): + with self.subTest('range-limited match'): + obj = re.compile(pattern) + self.assertTrue(obj.search(s, start, end + 1)) # Try the match with IGNORECASE enabled, and check that it # still succeeds. - obj = re.compile(pattern, re.IGNORECASE) - result = obj.search(s) - if result is None: - print('=== Fails on case-insensitive match', t) + with self.subTest('case-insensitive match'): + obj = re.compile(pattern, re.IGNORECASE) + self.assertTrue(obj.search(s)) # Try the match with LOCALE enabled, and check that it # still succeeds. if '(?u)' not in pattern: - obj = re.compile(pattern, re.LOCALE) - result = obj.search(s) - if result is None: - print('=== Fails on locale-sensitive match', t) + with self.subTest('locale-sensitive match'): + obj = re.compile(pattern, re.LOCALE) + self.assertTrue(obj.search(s)) # Try the match with UNICODE locale enabled, and check # that it still succeeds. - obj = re.compile(pattern, re.UNICODE) - result = obj.search(s) - if result is None: - print('=== Fails on unicode-sensitive match', t) + with self.subTest('unicode-sensitive match'): + obj = re.compile(pattern, re.UNICODE) + self.assertTrue(obj.search(s)) -def test_main(): - run_unittest(__name__) - run_re_tests() - if __name__ == "__main__": - test_main() + unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1348,6 +1348,8 @@ Tests ----- +- Issue #22838: All test_re tests now work with unittest test discovery. + - Issue #22173: Update lib2to3 tests to use unittest test discovery. - Issue #16000: Convert test_curses to use unittest. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 10:10:43 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 09:10:43 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyODM4?= =?utf-8?q?=3A_All_test=5Fre_tests_now_work_with_unittest_test_discovery?= =?utf-8?q?=2E?= Message-ID: <20141201091040.69791.99449@psf.io> https://hg.python.org/cpython/rev/21a7a92f4d0c changeset: 93673:21a7a92f4d0c branch: 3.4 parent: 93671:f5eb62bdcb1a user: Serhiy Storchaka date: Mon Dec 01 11:06:45 2014 +0200 summary: Issue #22838: All test_re tests now work with unittest test discovery. files: Lib/test/re_tests.py | 2 +- Lib/test/test_re.py | 142 +++++++++++++----------------- Misc/NEWS | 2 + 3 files changed, 64 insertions(+), 82 deletions(-) diff --git a/Lib/test/re_tests.py b/Lib/test/re_tests.py --- a/Lib/test/re_tests.py +++ b/Lib/test/re_tests.py @@ -86,7 +86,7 @@ (r'\a[\b]\f\n\r\t\v', '\a\b\f\n\r\t\v', SUCCEED, 'found', '\a\b\f\n\r\t\v'), (r'[\a][\b][\f][\n][\r][\t][\v]', '\a\b\f\n\r\t\v', SUCCEED, 'found', '\a\b\f\n\r\t\v'), # NOTE: not an error under PCRE/PRE: - # (r'\u', '', SYNTAX_ERROR), # A Perl escape + (r'\u', '', SYNTAX_ERROR), # A Perl escape (r'\c\e\g\h\i\j\k\m\o\p\q\y\z', 'ceghijkmopqyz', SUCCEED, 'found', 'ceghijkmopqyz'), (r'\xff', '\377', SUCCEED, 'found', chr(255)), # new \x semantics diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1439,55 +1439,55 @@ self.assertEqual(f("abcabdac"), [0, 0, 0, 1, 2, 0, 1, 0]) -def run_re_tests(): - from test.re_tests import tests, SUCCEED, FAIL, SYNTAX_ERROR - if verbose: - print('Running re_tests test suite') - else: - # To save time, only run the first and last 10 tests - #tests = tests[:10] + tests[-10:] - pass +class ExternalTests(unittest.TestCase): - for t in tests: - sys.stdout.flush() - pattern = s = outcome = repl = expected = None - if len(t) == 5: - pattern, s, outcome, repl, expected = t - elif len(t) == 3: - pattern, s, outcome = t - else: - raise ValueError('Test tuples should have 3 or 5 fields', t) + def test_re_benchmarks(self): + 're_tests benchmarks' + from test.re_tests import benchmarks + for pattern, s in benchmarks: + with self.subTest(pattern=pattern, string=s): + p = re.compile(pattern) + self.assertTrue(p.search(s)) + self.assertTrue(p.match(s)) + self.assertTrue(p.fullmatch(s)) + s2 = ' '*10000 + s + ' '*10000 + self.assertTrue(p.search(s2)) + self.assertTrue(p.match(s2, 10000)) + self.assertTrue(p.match(s2, 10000, 10000 + len(s))) + self.assertTrue(p.fullmatch(s2, 10000, 10000 + len(s))) - try: - obj = re.compile(pattern) - except re.error: - if outcome == SYNTAX_ERROR: pass # Expected a syntax error + def test_re_tests(self): + 're_tests test suite' + from test.re_tests import tests, SUCCEED, FAIL, SYNTAX_ERROR + for t in tests: + pattern = s = outcome = repl = expected = None + if len(t) == 5: + pattern, s, outcome, repl, expected = t + elif len(t) == 3: + pattern, s, outcome = t else: - print('=== Syntax error:', t) - except KeyboardInterrupt: raise KeyboardInterrupt - except: - print('*** Unexpected error ***', t) - if verbose: - traceback.print_exc(file=sys.stdout) - else: - try: + raise ValueError('Test tuples should have 3 or 5 fields', t) + + with self.subTest(pattern=pattern, string=s): + if outcome == SYNTAX_ERROR: # Expected a syntax error + with self.assertRaises(re.error): + re.compile(pattern) + continue + + obj = re.compile(pattern) result = obj.search(s) - except re.error as msg: - print('=== Unexpected exception', t, repr(msg)) - if outcome == SYNTAX_ERROR: - # This should have been a syntax error; forget it. - pass - elif outcome == FAIL: - if result is None: pass # No match, as expected - else: print('=== Succeeded incorrectly', t) - elif outcome == SUCCEED: - if result is not None: + if outcome == FAIL: + self.assertIsNone(result, 'Succeeded incorrectly') + continue + + with self.subTest(): + self.assertTrue(result, 'Failed incorrectly') # Matched, as expected, so now we compute the # result string and compare it to our expected result. start, end = result.span(0) - vardict={'found': result.group(0), - 'groups': result.group(), - 'flags': result.re.flags} + vardict = {'found': result.group(0), + 'groups': result.group(), + 'flags': result.re.flags} for i in range(1, 100): try: gi = result.group(i) @@ -1505,12 +1505,8 @@ except IndexError: gi = "Error" vardict[i] = gi - repl = eval(repl, vardict) - if repl != expected: - print('=== grouping error', t, end=' ') - print(repr(repl) + ' should be ' + repr(expected)) - else: - print('=== Failed incorrectly', t) + self.assertEqual(eval(repl, vardict), expected, + 'grouping error') # Try the match with both pattern and string converted to # bytes, and check that it still succeeds. @@ -1521,55 +1517,39 @@ # skip non-ascii tests pass else: - try: + with self.subTest('bytes pattern match'): bpat = re.compile(bpat) - except Exception: - print('=== Fails on bytes pattern compile', t) - if verbose: - traceback.print_exc(file=sys.stdout) - else: - bytes_result = bpat.search(bs) - if bytes_result is None: - print('=== Fails on bytes pattern match', t) + self.assertTrue(bpat.search(bs)) # Try the match with the search area limited to the extent # of the match and see if it still succeeds. \B will # break (because it won't match at the end or start of a # string), so we'll ignore patterns that feature it. - - if pattern[:2] != '\\B' and pattern[-2:] != '\\B' \ - and result is not None: - obj = re.compile(pattern) - result = obj.search(s, result.start(0), result.end(0) + 1) - if result is None: - print('=== Failed on range-limited match', t) + if (pattern[:2] != r'\B' and pattern[-2:] != r'\B' + and result is not None): + with self.subTest('range-limited match'): + obj = re.compile(pattern) + self.assertTrue(obj.search(s, start, end + 1)) # Try the match with IGNORECASE enabled, and check that it # still succeeds. - obj = re.compile(pattern, re.IGNORECASE) - result = obj.search(s) - if result is None: - print('=== Fails on case-insensitive match', t) + with self.subTest('case-insensitive match'): + obj = re.compile(pattern, re.IGNORECASE) + self.assertTrue(obj.search(s)) # Try the match with LOCALE enabled, and check that it # still succeeds. if '(?u)' not in pattern: - obj = re.compile(pattern, re.LOCALE) - result = obj.search(s) - if result is None: - print('=== Fails on locale-sensitive match', t) + with self.subTest('locale-sensitive match'): + obj = re.compile(pattern, re.LOCALE) + self.assertTrue(obj.search(s)) # Try the match with UNICODE locale enabled, and check # that it still succeeds. - obj = re.compile(pattern, re.UNICODE) - result = obj.search(s) - if result is None: - print('=== Fails on unicode-sensitive match', t) + with self.subTest('unicode-sensitive match'): + obj = re.compile(pattern, re.UNICODE) + self.assertTrue(obj.search(s)) -def test_main(): - run_unittest(__name__) - run_re_tests() - if __name__ == "__main__": - test_main() + unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -158,6 +158,8 @@ Tests ----- +- Issue #22838: All test_re tests now work with unittest test discovery. + - Issue #22173: Update lib2to3 tests to use unittest test discovery. - Issue #16000: Convert test_curses to use unittest. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 10:27:18 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 09:27:18 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Removed_unused_function_li?= =?utf-8?q?necol=28=29_=28added_in_issue_=2322578_by_mistake=29=2E?= Message-ID: <20141201092716.69805.77126@psf.io> https://hg.python.org/cpython/rev/9711c60e3049 changeset: 93675:9711c60e3049 user: Serhiy Storchaka date: Mon Dec 01 11:25:21 2014 +0200 summary: Removed unused function linecol() (added in issue #22578 by mistake). files: Lib/sre_constants.py | 12 ------------ 1 files changed, 0 insertions(+), 12 deletions(-) diff --git a/Lib/sre_constants.py b/Lib/sre_constants.py --- a/Lib/sre_constants.py +++ b/Lib/sre_constants.py @@ -39,18 +39,6 @@ self.lineno = self.colno = None super().__init__(msg) -def linecol(doc, pos): - if isinstance(pattern, str): - newline = '\n' - else: - newline = b'\n' - lineno = pattern.count(newline, 0, pos) + 1 - if lineno == 1: - colno = pos + 1 - else: - colno = pos - doc.rindex(newline, 0, pos) - return lineno, colno - class _NamedIntConstant(int): def __new__(cls, value, name): -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Mon Dec 1 10:35:21 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 01 Dec 2014 10:35:21 +0100 Subject: [Python-checkins] Daily reference leaks (64bb01bce12c): sum=3 Message-ID: results for 64bb01bce12c on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogW4W7eJ', '-x'] From python-checkins at python.org Mon Dec 1 10:52:10 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 09:52:10 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321032=3A_Deprecat?= =?utf-8?q?ed_the_use_of_re=2ELOCALE_flag_with_str_patterns_or?= Message-ID: <20141201095209.116318.69935@psf.io> https://hg.python.org/cpython/rev/561d1d0de518 changeset: 93676:561d1d0de518 user: Serhiy Storchaka date: Mon Dec 01 11:50:07 2014 +0200 summary: Issue #21032: Deprecated the use of re.LOCALE flag with str patterns or re.ASCII. It was newer worked. files: Doc/library/re.rst | 6 ++- Lib/sre_parse.py | 10 ++++ Lib/test/test_re.py | 82 ++++++++++++++++++++++++++------ Misc/NEWS | 3 + 4 files changed, 84 insertions(+), 17 deletions(-) diff --git a/Doc/library/re.rst b/Doc/library/re.rst --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -521,7 +521,11 @@ current locale. The use of this flag is discouraged as the locale mechanism is very unreliable, and it only handles one "culture" at a time anyway; you should use Unicode matching instead, which is the default in Python 3 - for Unicode (str) patterns. + for Unicode (str) patterns. This flag makes sense only with bytes patterns. + + .. deprecated-removed:: 3.5 3.6 + Deprecated the use of :const:`re.LOCALE` with string patterns or + :const:`re.ASCII`. .. data:: M diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -751,6 +751,11 @@ def fix_flags(src, flags): # Check and fix flags according to the type of pattern (str or bytes) if isinstance(src, str): + if flags & SRE_FLAG_LOCALE: + import warnings + warnings.warn("LOCALE flag with a str pattern is deprecated. " + "Will be an error in 3.6", + DeprecationWarning, stacklevel=6) if not flags & SRE_FLAG_ASCII: flags |= SRE_FLAG_UNICODE elif flags & SRE_FLAG_UNICODE: @@ -758,6 +763,11 @@ else: if flags & SRE_FLAG_UNICODE: raise ValueError("can't use UNICODE flag with a bytes pattern") + if flags & SRE_FLAG_LOCALE and flags & SRE_FLAG_ASCII: + import warnings + warnings.warn("ASCII and LOCALE flags are incompatible. " + "Will be an error in 3.6", + DeprecationWarning, stacklevel=6) return flags def parse(str, flags=0, pattern=None): diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -502,10 +502,6 @@ "abcd abc bcd bx", re.ASCII).group(1), "bx") self.assertEqual(re.search(r"\B(b.)\B", "abc bcd bc abxd", re.ASCII).group(1), "bx") - self.assertEqual(re.search(r"\b(b.)\b", - "abcd abc bcd bx", re.LOCALE).group(1), "bx") - self.assertEqual(re.search(r"\B(b.)\B", - "abc bcd bc abxd", re.LOCALE).group(1), "bx") self.assertEqual(re.search(r"^abc$", "\nabc\n", re.M).group(0), "abc") self.assertEqual(re.search(r"^\Aabc\Z$", "abc", re.M).group(0), "abc") self.assertIsNone(re.search(r"^\Aabc\Z$", "\nabc\n", re.M)) @@ -526,8 +522,6 @@ b"1aa! a").group(0), b"1aa! a") self.assertEqual(re.search(r"\d\D\w\W\s\S", "1aa! a", re.ASCII).group(0), "1aa! a") - self.assertEqual(re.search(r"\d\D\w\W\s\S", - "1aa! a", re.LOCALE).group(0), "1aa! a") self.assertEqual(re.search(br"\d\D\w\W\s\S", b"1aa! a", re.LOCALE).group(0), b"1aa! a") @@ -693,9 +687,12 @@ self.assertEqual(_sre.getlower(ord('A'), 0), ord('a')) self.assertEqual(_sre.getlower(ord('A'), re.LOCALE), ord('a')) self.assertEqual(_sre.getlower(ord('A'), re.UNICODE), ord('a')) + self.assertEqual(_sre.getlower(ord('A'), re.ASCII), ord('a')) self.assertEqual(re.match("abc", "ABC", re.I).group(0), "ABC") self.assertEqual(re.match(b"abc", b"ABC", re.I).group(0), b"ABC") + self.assertEqual(re.match("abc", "ABC", re.I|re.A).group(0), "ABC") + self.assertEqual(re.match(b"abc", b"ABC", re.I|re.L).group(0), b"ABC") def test_not_literal(self): self.assertEqual(re.search("\s([^a])", " b").group(1), "b") @@ -780,8 +777,10 @@ self.assertEqual(re.X, re.VERBOSE) def test_flags(self): - for flag in [re.I, re.M, re.X, re.S, re.L]: + for flag in [re.I, re.M, re.X, re.S, re.A, re.U]: self.assertTrue(re.compile('^pattern$', flag)) + for flag in [re.I, re.M, re.X, re.S, re.A, re.L]: + self.assertTrue(re.compile(b'^pattern$', flag)) def test_sre_character_literals(self): for i in [0, 8, 16, 32, 64, 127, 128, 255, 256, 0xFFFF, 0x10000, 0x10FFFF]: @@ -1146,6 +1145,52 @@ self.assertRaises(ValueError, re.compile, '(?a)\w', re.UNICODE) self.assertRaises(ValueError, re.compile, '(?au)\w') + def test_locale_flag(self): + import locale + _, enc = locale.getlocale(locale.LC_CTYPE) + # Search non-ASCII letter + for i in range(128, 256): + try: + c = bytes([i]).decode(enc) + sletter = c.lower() + if sletter == c: continue + bletter = sletter.encode(enc) + if len(bletter) != 1: continue + if bletter.decode(enc) != sletter: continue + bpat = re.escape(bytes([i])) + break + except (UnicodeError, TypeError): + pass + else: + bletter = None + bpat = b'A' + # Bytes patterns + pat = re.compile(bpat, re.LOCALE | re.IGNORECASE) + if bletter: + self.assertTrue(pat.match(bletter)) + pat = re.compile(b'(?L)' + bpat, re.IGNORECASE) + if bletter: + self.assertTrue(pat.match(bletter)) + pat = re.compile(bpat, re.IGNORECASE) + if bletter: + self.assertIsNone(pat.match(bletter)) + pat = re.compile(b'\w', re.LOCALE) + if bletter: + self.assertTrue(pat.match(bletter)) + pat = re.compile(b'(?L)\w') + if bletter: + self.assertTrue(pat.match(bletter)) + pat = re.compile(b'\w') + if bletter: + self.assertIsNone(pat.match(bletter)) + # Incompatibilities + self.assertWarns(DeprecationWarning, re.compile, '', re.LOCALE) + self.assertWarns(DeprecationWarning, re.compile, '(?L)') + self.assertWarns(DeprecationWarning, re.compile, b'', re.LOCALE | re.ASCII) + self.assertWarns(DeprecationWarning, re.compile, b'(?L)', re.ASCII) + self.assertWarns(DeprecationWarning, re.compile, b'(?a)', re.LOCALE) + self.assertWarns(DeprecationWarning, re.compile, b'(?aL)') + def test_bug_6509(self): # Replacement strings of both types must parse properly. # all strings @@ -1477,6 +1522,10 @@ self.check_flags(b'bytes pattern', re.A, "re.compile(b'bytes pattern', re.ASCII)") + def test_locale(self): + self.check_flags(b'bytes pattern', re.L, + "re.compile(b'bytes pattern', re.LOCALE)") + def test_quotes(self): self.check('random "double quoted" pattern', '''re.compile('random "double quoted" pattern')''') @@ -1590,8 +1639,16 @@ pass else: with self.subTest('bytes pattern match'): - bpat = re.compile(bpat) - self.assertTrue(bpat.search(bs)) + obj = re.compile(bpat) + self.assertTrue(obj.search(bs)) + + # Try the match with LOCALE enabled, and check that it + # still succeeds. + with self.subTest('locale-sensitive match'): + obj = re.compile(bpat, re.LOCALE) + result = obj.search(bs) + if result is None: + print('=== Fails on locale-sensitive match', t) # Try the match with the search area limited to the extent # of the match and see if it still succeeds. \B will @@ -1609,13 +1666,6 @@ obj = re.compile(pattern, re.IGNORECASE) self.assertTrue(obj.search(s)) - # Try the match with LOCALE enabled, and check that it - # still succeeds. - if '(?u)' not in pattern: - with self.subTest('locale-sensitive match'): - obj = re.compile(pattern, re.LOCALE) - self.assertTrue(obj.search(s)) - # Try the match with UNICODE locale enabled, and check # that it still succeeds. with self.subTest('unicode-sensitive match'): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -191,6 +191,9 @@ Library ------- +- Issue #21032: Deprecated the use of re.LOCALE flag with str patterns or + re.ASCII. It was newer worked. + - Issue #22902: The "ip" command is now used on Linux to determine MAC address in uuid.getnode(). Pach by Bruno Cauet. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 12:16:35 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 11:16:35 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fixed_issue_number_in_Misc?= =?utf-8?q?/NEWS_for_issue_=2322407=2E?= Message-ID: <20141201111629.116326.68978@psf.io> https://hg.python.org/cpython/rev/abc7fe393016 changeset: 93677:abc7fe393016 user: Serhiy Storchaka date: Mon Dec 01 12:47:16 2014 +0200 summary: Fixed issue number in Misc/NEWS for issue #22407. files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -191,7 +191,7 @@ Library ------- -- Issue #21032: Deprecated the use of re.LOCALE flag with str patterns or +- Issue #22407: Deprecated the use of re.LOCALE flag with str patterns or re.ASCII. It was newer worked. - Issue #22902: The "ip" command is now used on Linux to determine MAC address -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 12:16:36 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 11:16:36 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMDMy?= =?utf-8?q?=2E_Fixed_socket_leak_if_HTTPConnection=2Egetresponse=28=29_fai?= =?utf-8?b?bHMu?= Message-ID: <20141201111629.116304.13037@psf.io> https://hg.python.org/cpython/rev/f88c00391dd8 changeset: 93678:f88c00391dd8 branch: 2.7 parent: 93670:df17d2b0878f user: Serhiy Storchaka date: Mon Dec 01 13:07:28 2014 +0200 summary: Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. Original patch by Martin Panter. files: Lib/httplib.py | 24 ++++++++++++++---------- Lib/test/test_httplib.py | 25 ++++++++++++++++++++++++- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 4 files changed, 42 insertions(+), 11 deletions(-) diff --git a/Lib/httplib.py b/Lib/httplib.py --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -1070,18 +1070,22 @@ kwds["buffering"] = True; response = self.response_class(*args, **kwds) - response.begin() - assert response.will_close != _UNKNOWN - self.__state = _CS_IDLE + try: + response.begin() + assert response.will_close != _UNKNOWN + self.__state = _CS_IDLE - if response.will_close: - # this effectively passes the connection to the response - self.close() - else: - # remember this, so we can tell when it is complete - self.__response = response + if response.will_close: + # this effectively passes the connection to the response + self.close() + else: + # remember this, so we can tell when it is complete + self.__response = response - return response + return response + except: + response.close() + raise class HTTP: diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -25,6 +25,7 @@ self.text = text self.fileclass = fileclass self.data = '' + self.file_closed = False self.host = host self.port = port @@ -34,7 +35,13 @@ def makefile(self, mode, bufsize=None): if mode != 'r' and mode != 'rb': raise httplib.UnimplementedFileMode() - return self.fileclass(self.text) + # keep the file around so we can check how much was read from it + self.file = self.fileclass(self.text) + self.file.close = self.file_close #nerf close () + return self.file + + def file_close(self): + self.file_closed = True def close(self): pass @@ -433,6 +440,22 @@ self.assertEqual(resp.read(), '') self.assertTrue(resp.isclosed()) + def test_error_leak(self): + # Test that the socket is not leaked if getresponse() fails + conn = httplib.HTTPConnection('example.com') + response = [] + class Response(httplib.HTTPResponse): + def __init__(self, *pos, **kw): + response.append(self) # Avoid garbage collector closing the socket + httplib.HTTPResponse.__init__(self, *pos, **kw) + conn.response_class = Response + conn.sock = FakeSocket('') # Emulate server dropping connection + conn.request('GET', '/') + self.assertRaises(httplib.BadStatusLine, conn.getresponse) + self.assertTrue(response) + #self.assertTrue(response[0].closed) + self.assertTrue(conn.sock.file_closed) + class OfflineTest(TestCase): def test_responses(self): self.assertEqual(httplib.responses[httplib.NOT_FOUND], "Not Found") diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1010,6 +1010,7 @@ Juan David Ib??ez Palomar Jan Palus Yongzhi Pan +Martin Panter Mathias Panzenb?ck M. Papillon Peter Parente diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,9 @@ Library ------- +- Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. + Original patch by Martin Panter. + - Issue #22609: Constructors and update methods of mapping classes in the collections module now accept the self keyword argument. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 12:16:37 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 11:16:37 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2321032=2E_Fixed_socket_leak_if_HTTPConnection=2E?= =?utf-8?q?getresponse=28=29_fails=2E?= Message-ID: <20141201111630.126788.71491@psf.io> https://hg.python.org/cpython/rev/62a058c76869 changeset: 93680:62a058c76869 parent: 93677:abc7fe393016 parent: 93679:ba72da4883eb user: Serhiy Storchaka date: Mon Dec 01 13:10:12 2014 +0200 summary: Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. Original patch by Martin Panter. files: Lib/http/client.py | 24 ++++++++++++++---------- Lib/test/test_httplib.py | 22 +++++++++++++++++++++- Misc/NEWS | 3 +++ 3 files changed, 38 insertions(+), 11 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -1233,18 +1233,22 @@ else: response = self.response_class(self.sock, method=self._method) - response.begin() - assert response.will_close != _UNKNOWN - self.__state = _CS_IDLE + try: + response.begin() + assert response.will_close != _UNKNOWN + self.__state = _CS_IDLE - if response.will_close: - # this effectively passes the connection to the response - self.close() - else: - # remember this, so we can tell when it is complete - self.__response = response + if response.will_close: + # this effectively passes the connection to the response + self.close() + else: + # remember this, so we can tell when it is complete + self.__response = response - return response + return response + except: + response.close() + raise try: import ssl diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -48,6 +48,7 @@ self.fileclass = fileclass self.data = b'' self.sendall_calls = 0 + self.file_closed = False self.host = host self.port = port @@ -60,9 +61,12 @@ raise client.UnimplementedFileMode() # keep the file around so we can check how much was read from it self.file = self.fileclass(self.text) - self.file.close = lambda:None #nerf close () + self.file.close = self.file_close #nerf close () return self.file + def file_close(self): + self.file_closed = True + def close(self): pass @@ -676,6 +680,22 @@ conn.request('POST', '/', body) self.assertGreater(sock.sendall_calls, 1) + def test_error_leak(self): + # Test that the socket is not leaked if getresponse() fails + conn = client.HTTPConnection('example.com') + response = None + class Response(client.HTTPResponse): + def __init__(self, *pos, **kw): + nonlocal response + response = self # Avoid garbage collector closing the socket + client.HTTPResponse.__init__(self, *pos, **kw) + conn.response_class = Response + conn.sock = FakeSocket('') # Emulate server dropping connection + conn.request('GET', '/') + self.assertRaises(client.BadStatusLine, conn.getresponse) + self.assertTrue(response.closed) + self.assertTrue(conn.sock.file_closed) + def test_chunked_extension(self): extra = '3;foo=bar\r\n' + 'abc\r\n' expected = chunked_expected + b'abc' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -191,6 +191,9 @@ Library ------- +- Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. + Original patch by Martin Panter. + - Issue #22407: Deprecated the use of re.LOCALE flag with str patterns or re.ASCII. It was newer worked. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 12:16:38 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 11:16:38 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDMy?= =?utf-8?q?=2E_Fixed_socket_leak_if_HTTPConnection=2Egetresponse=28=29_fai?= =?utf-8?b?bHMu?= Message-ID: <20141201111629.84301.16696@psf.io> https://hg.python.org/cpython/rev/ba72da4883eb changeset: 93679:ba72da4883eb branch: 3.4 parent: 93673:21a7a92f4d0c user: Serhiy Storchaka date: Mon Dec 01 13:07:45 2014 +0200 summary: Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. Original patch by Martin Panter. files: Lib/http/client.py | 24 ++++++++++++++---------- Lib/test/test_httplib.py | 25 ++++++++++++++++++++++++- Misc/NEWS | 3 +++ 3 files changed, 41 insertions(+), 11 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -1169,18 +1169,22 @@ else: response = self.response_class(self.sock, method=self._method) - response.begin() - assert response.will_close != _UNKNOWN - self.__state = _CS_IDLE + try: + response.begin() + assert response.will_close != _UNKNOWN + self.__state = _CS_IDLE - if response.will_close: - # this effectively passes the connection to the response - self.close() - else: - # remember this, so we can tell when it is complete - self.__response = response + if response.will_close: + # this effectively passes the connection to the response + self.close() + else: + # remember this, so we can tell when it is complete + self.__response = response - return response + return response + except: + response.close() + raise try: import ssl diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -28,6 +28,7 @@ self.fileclass = fileclass self.data = b'' self.sendall_calls = 0 + self.file_closed = False self.host = host self.port = port @@ -38,7 +39,13 @@ def makefile(self, mode, bufsize=None): if mode != 'r' and mode != 'rb': raise client.UnimplementedFileMode() - return self.fileclass(self.text) + # keep the file around so we can check how much was read from it + self.file = self.fileclass(self.text) + self.file.close = self.file_close #nerf close () + return self.file + + def file_close(self): + self.file_closed = True def close(self): pass @@ -675,6 +682,22 @@ conn.request('POST', '/', body) self.assertGreater(sock.sendall_calls, 1) + def test_error_leak(self): + # Test that the socket is not leaked if getresponse() fails + conn = client.HTTPConnection('example.com') + response = None + class Response(client.HTTPResponse): + def __init__(self, *pos, **kw): + nonlocal response + response = self # Avoid garbage collector closing the socket + client.HTTPResponse.__init__(self, *pos, **kw) + conn.response_class = Response + conn.sock = FakeSocket('') # Emulate server dropping connection + conn.request('GET', '/') + self.assertRaises(client.BadStatusLine, conn.getresponse) + self.assertTrue(response.closed) + self.assertTrue(conn.sock.file_closed) + class OfflineTest(TestCase): def test_responses(self): self.assertEqual(client.responses[client.NOT_FOUND], "Not Found") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -36,6 +36,9 @@ Library ------- +- Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. + Original patch by Martin Panter. + - Issue #22960: Add a context argument to xmlrpclib.ServerProxy constructor. - Issue #22915: SAX parser now supports files opened with file descriptor or -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 17:34:54 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 16:34:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Removed_duplic?= =?utf-8?q?ated_words_in_in_comments_and_docs=2E?= Message-ID: <20141201163438.84279.97637@psf.io> https://hg.python.org/cpython/rev/cc744bf70cf6 changeset: 93681:cc744bf70cf6 branch: 2.7 parent: 93678:f88c00391dd8 user: Serhiy Storchaka date: Mon Dec 01 18:16:30 2014 +0200 summary: Removed duplicated words in in comments and docs. files: Lib/hashlib.py | 2 +- Lib/test/test_docxmlrpc.py | 2 +- Modules/posixmodule.c | 2 +- Python/pystrtod.c | 2 +- Python/random.c | 2 +- Python/thread.c | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Lib/hashlib.py b/Lib/hashlib.py --- a/Lib/hashlib.py +++ b/Lib/hashlib.py @@ -187,7 +187,7 @@ def prf(msg, inner=inner, outer=outer): # PBKDF2_HMAC uses the password as key. We can re-use the same - # digest objects and and just update copies to skip initialization. + # digest objects and just update copies to skip initialization. icpy = inner.copy() ocpy = outer.copy() icpy.update(msg) diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -10,7 +10,7 @@ PORT = None def make_request_and_skipIf(condition, reason): - # If we skip the test, we have to make a request because the + # If we skip the test, we have to make a request because # the server created in setUp blocks expecting one to come in. if not condition: return lambda func: func diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -8496,7 +8496,7 @@ /* This code is used to ensure that the tables of configuration value names - * are in sorted order as required by conv_confname(), and also to build the + * are in sorted order as required by conv_confname(), and also to build * the exported dictionaries that are used to publish information about the * names available on the host platform. * diff --git a/Python/pystrtod.c b/Python/pystrtod.c --- a/Python/pystrtod.c +++ b/Python/pystrtod.c @@ -324,7 +324,7 @@ On overflow (e.g., when trying to convert '1e500' on an IEEE 754 machine), if overflow_exception is NULL then +-Py_HUGE_VAL is returned, and no Python - exception is raised. Otherwise, overflow_exception should point to a + exception is raised. Otherwise, overflow_exception should point to a Python exception, this exception will be raised, -1.0 will be returned, and *endptr will point just past the end of the converted value. diff --git a/Python/random.c b/Python/random.c --- a/Python/random.c +++ b/Python/random.c @@ -274,7 +274,7 @@ } /* Fill buffer with size pseudo-random bytes from the operating system random - number generator (RNG). It is suitable for for most cryptographic purposes + number generator (RNG). It is suitable for most cryptographic purposes except long living private keys for asymmetric encryption. Return 0 on success, raise an exception and return -1 on error. */ diff --git a/Python/thread.c b/Python/thread.c --- a/Python/thread.c +++ b/Python/thread.c @@ -271,7 +271,7 @@ if (p->id == id && p->key == key) goto Done; /* Sanity check. These states should never happen but if - * they do we must abort. Otherwise we'll end up spinning in + * they do we must abort. Otherwise we'll end up spinning * in a tight loop with the lock held. A similar check is done * in pystate.c tstate_delete_common(). */ if (p == prev_p) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 17:35:00 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 16:35:00 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Removed_duplicated_words_in_in_comments_and_docs=2E?= Message-ID: <20141201163439.69795.59681@psf.io> https://hg.python.org/cpython/rev/506d22434d63 changeset: 93683:506d22434d63 parent: 93680:62a058c76869 parent: 93682:a4f3e78f68e4 user: Serhiy Storchaka date: Mon Dec 01 18:30:14 2014 +0200 summary: Removed duplicated words in in comments and docs. files: Doc/library/base64.rst | 2 +- Doc/whatsnew/3.3.rst | 2 +- Lib/asyncio/futures.py | 2 +- Lib/hashlib.py | 2 +- Lib/test/test_asyncio/test_windows_events.py | 2 +- Lib/test/test_docxmlrpc.py | 2 +- Lib/test/test_xmlrpc.py | 2 +- Lib/test/test_zipimport_support.py | 2 +- Lib/xml/etree/ElementTree.py | 2 +- Misc/HISTORY | 4 ++-- Misc/NEWS | 6 +++--- Modules/_tracemalloc.c | 2 +- Modules/posixmodule.c | 2 +- Python/pystrtod.c | 2 +- Python/random.c | 2 +- Python/thread.c | 2 +- 16 files changed, 19 insertions(+), 19 deletions(-) diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -13,7 +13,7 @@ This module provides functions for encoding binary data to printable ASCII characters and decoding such encodings back to binary data. It provides encoding and decoding functions for the encodings specified in -in :rfc:`3548`, which defines the Base16, Base32, and Base64 algorithms, +:rfc:`3548`, which defines the Base16, Base32, and Base64 algorithms, and for the de-facto standard Ascii85 and Base85 encodings. The :rfc:`3548` encodings are suitable for encoding binary data so that it can diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1540,7 +1540,7 @@ to override the default behavior of inheriting the ``daemon`` flag from the parent process (:issue:`6064`). -New attribute attribute :data:`multiprocessing.Process.sentinel` allows a +New attribute :data:`multiprocessing.Process.sentinel` allows a program to wait on multiple :class:`~multiprocessing.Process` objects at one time using the appropriate OS primitives (for example, :mod:`select` on posix systems). diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -61,7 +61,7 @@ the Future is collected, and the helper is present, the helper object is also collected, and its __del__() method will log the traceback. When the Future's result() or exception() method is - called (and a helper object is present), it removes the the helper + called (and a helper object is present), it removes the helper object, after calling its clear() method to prevent it from logging. diff --git a/Lib/hashlib.py b/Lib/hashlib.py --- a/Lib/hashlib.py +++ b/Lib/hashlib.py @@ -172,7 +172,7 @@ def prf(msg, inner=inner, outer=outer): # PBKDF2_HMAC uses the password as key. We can re-use the same - # digest objects and and just update copies to skip initialization. + # digest objects and just update copies to skip initialization. icpy = inner.copy() ocpy = outer.copy() icpy.update(msg) diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -105,7 +105,7 @@ _overlapped.SetEvent(event) - # Wait for for set event; + # Wait for set event; # result should be True immediately fut = self.loop._proactor.wait_for_handle(event, 10) start = self.loop.time() diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -10,7 +10,7 @@ PORT = None def make_request_and_skipIf(condition, reason): - # If we skip the test, we have to make a request because the + # If we skip the test, we have to make a request because # the server created in setUp blocks expecting one to come in. if not condition: return lambda func: func diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -517,7 +517,7 @@ return True def make_request_and_skipIf(condition, reason): - # If we skip the test, we have to make a request because the + # If we skip the test, we have to make a request because # the server created in setUp blocks expecting one to come in. if not condition: return lambda func: func diff --git a/Lib/test/test_zipimport_support.py b/Lib/test/test_zipimport_support.py --- a/Lib/test/test_zipimport_support.py +++ b/Lib/test/test_zipimport_support.py @@ -57,7 +57,7 @@ # This used to use the ImportHooksBaseTestCase to restore # the state of the import related information # in the sys module after each test. However, that restores - # *too much* information and breaks for the invocation of + # *too much* information and breaks for the invocation # of test_doctest. So we do our own thing and leave # sys.modules alone. # We also clear the linecache and zipimport cache diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -125,7 +125,7 @@ This class is the reference implementation of the Element interface. An element's length is its number of subelements. That means if you - you want to check if an element is truly empty, you should check BOTH + want to check if an element is truly empty, you should check BOTH its length AND its text attribute. The element tag, attribute names, and attribute values can be either diff --git a/Misc/HISTORY b/Misc/HISTORY --- a/Misc/HISTORY +++ b/Misc/HISTORY @@ -3970,7 +3970,7 @@ - Issue #12041: Make test_wait3 more robust. - Issue #11873: Change regex in test_compileall to fix occasional failures when - when the randomly generated temporary path happened to match the regex. + the randomly generated temporary path happened to match the regex. - Issue #11958: Fix FTP tests for IPv6, bind to "::1" instead of "localhost". Patch written by Charles-Francois Natali. @@ -14051,7 +14051,7 @@ - When method objects have an attribute that can be satisfied either by the function object or by the method object, the function object's attribute usually wins. Christian Tismer pointed out that - that this is really a mistake, because this only happens for special + this is really a mistake, because this only happens for special methods (like __reduce__) where the method object's version is really more appropriate than the function's attribute. So from now on, all method attributes will have precedence over function diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1813,7 +1813,7 @@ - Issue #15304: concurrent.futures.wait() can block forever even if Futures have completed. Patch by Glenn Langford. -- Issue #14455: plistlib: fix serializing integers integers in the range +- Issue #14455: plistlib: fix serializing integers in the range of an unsigned long long but outside of the range of signed long long for binary plist files. @@ -1824,7 +1824,7 @@ Patch mostly by Serhiy Storchaka. - Update the python.gif icon for the Idle classbrowser and pathbowser - from the old green snake to the new new blue and yellow snakes. + from the old green snake to the new blue and yellow snakes. - Issue #17721: Remove non-functional configuration dialog help button until we make it actually gives some help when clicked. Patch by Guilherme Sim?es. @@ -3105,7 +3105,7 @@ reside in the os module. - Issue #19205: Don't import the 're' module in site and sysconfig module to - to speed up interpreter start. + speed up interpreter start. - Issue #9548: Add a minimal "_bootlocale" module that is imported by the _io module instead of the full locale module. diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -101,7 +101,7 @@ Protected by TABLES_LOCK(). */ static size_t tracemalloc_peak_traced_memory = 0; -/* Hash table used as a set to to intern filenames: +/* Hash table used as a set to intern filenames: PyObject* => PyObject*. Protected by the GIL */ static _Py_hashtable_t *tracemalloc_filenames = NULL; diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -14944,7 +14944,7 @@ /* This code is used to ensure that the tables of configuration value names - * are in sorted order as required by conv_confname(), and also to build the + * are in sorted order as required by conv_confname(), and also to build * the exported dictionaries that are used to publish information about the * names available on the host platform. * diff --git a/Python/pystrtod.c b/Python/pystrtod.c --- a/Python/pystrtod.c +++ b/Python/pystrtod.c @@ -325,7 +325,7 @@ On overflow (e.g., when trying to convert '1e500' on an IEEE 754 machine), if overflow_exception is NULL then +-Py_HUGE_VAL is returned, and no Python - exception is raised. Otherwise, overflow_exception should point to a + exception is raised. Otherwise, overflow_exception should point to a Python exception, this exception will be raised, -1.0 will be returned, and *endptr will point just past the end of the converted value. diff --git a/Python/random.c b/Python/random.c --- a/Python/random.c +++ b/Python/random.c @@ -225,7 +225,7 @@ } /* Fill buffer with size pseudo-random bytes from the operating system random - number generator (RNG). It is suitable for for most cryptographic purposes + number generator (RNG). It is suitable for most cryptographic purposes except long living private keys for asymmetric encryption. Return 0 on success, raise an exception and return -1 on error. */ diff --git a/Python/thread.c b/Python/thread.c --- a/Python/thread.c +++ b/Python/thread.c @@ -221,7 +221,7 @@ goto Done; } /* Sanity check. These states should never happen but if - * they do we must abort. Otherwise we'll end up spinning in + * they do we must abort. Otherwise we'll end up spinning * in a tight loop with the lock held. A similar check is done * in pystate.c tstate_delete_common(). */ if (p == prev_p) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 17:35:00 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 16:35:00 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Removed_duplic?= =?utf-8?q?ated_words_in_in_comments_and_docs=2E?= Message-ID: <20141201163438.126788.46543@psf.io> https://hg.python.org/cpython/rev/a4f3e78f68e4 changeset: 93682:a4f3e78f68e4 branch: 3.4 parent: 93679:ba72da4883eb user: Serhiy Storchaka date: Mon Dec 01 18:28:43 2014 +0200 summary: Removed duplicated words in in comments and docs. files: Doc/library/base64.rst | 2 +- Doc/whatsnew/3.3.rst | 2 +- Lib/asyncio/futures.py | 2 +- Lib/hashlib.py | 2 +- Lib/test/test_asyncio/test_windows_events.py | 2 +- Lib/test/test_docxmlrpc.py | 2 +- Lib/test/test_xmlrpc.py | 2 +- Lib/test/test_zipimport_support.py | 2 +- Lib/xml/etree/ElementTree.py | 2 +- Misc/HISTORY | 4 ++-- Misc/NEWS | 6 +++--- Modules/_tracemalloc.c | 2 +- Modules/posixmodule.c | 2 +- Python/pystrtod.c | 2 +- Python/random.c | 2 +- Python/thread.c | 2 +- 16 files changed, 19 insertions(+), 19 deletions(-) diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -13,7 +13,7 @@ This module provides functions for encoding binary data to printable ASCII characters and decoding such encodings back to binary data. It provides encoding and decoding functions for the encodings specified in -in :rfc:`3548`, which defines the Base16, Base32, and Base64 algorithms, +:rfc:`3548`, which defines the Base16, Base32, and Base64 algorithms, and for the de-facto standard Ascii85 and Base85 encodings. The :rfc:`3548` encodings are suitable for encoding binary data so that it can diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1540,7 +1540,7 @@ to override the default behavior of inheriting the ``daemon`` flag from the parent process (:issue:`6064`). -New attribute attribute :data:`multiprocessing.Process.sentinel` allows a +New attribute :data:`multiprocessing.Process.sentinel` allows a program to wait on multiple :class:`~multiprocessing.Process` objects at one time using the appropriate OS primitives (for example, :mod:`select` on posix systems). diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -61,7 +61,7 @@ the Future is collected, and the helper is present, the helper object is also collected, and its __del__() method will log the traceback. When the Future's result() or exception() method is - called (and a helper object is present), it removes the the helper + called (and a helper object is present), it removes the helper object, after calling its clear() method to prevent it from logging. diff --git a/Lib/hashlib.py b/Lib/hashlib.py --- a/Lib/hashlib.py +++ b/Lib/hashlib.py @@ -172,7 +172,7 @@ def prf(msg, inner=inner, outer=outer): # PBKDF2_HMAC uses the password as key. We can re-use the same - # digest objects and and just update copies to skip initialization. + # digest objects and just update copies to skip initialization. icpy = inner.copy() ocpy = outer.copy() icpy.update(msg) diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -105,7 +105,7 @@ _overlapped.SetEvent(event) - # Wait for for set event; + # Wait for set event; # result should be True immediately fut = self.loop._proactor.wait_for_handle(event, 10) start = self.loop.time() diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -10,7 +10,7 @@ PORT = None def make_request_and_skipIf(condition, reason): - # If we skip the test, we have to make a request because the + # If we skip the test, we have to make a request because # the server created in setUp blocks expecting one to come in. if not condition: return lambda func: func diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -517,7 +517,7 @@ return True def make_request_and_skipIf(condition, reason): - # If we skip the test, we have to make a request because the + # If we skip the test, we have to make a request because # the server created in setUp blocks expecting one to come in. if not condition: return lambda func: func diff --git a/Lib/test/test_zipimport_support.py b/Lib/test/test_zipimport_support.py --- a/Lib/test/test_zipimport_support.py +++ b/Lib/test/test_zipimport_support.py @@ -57,7 +57,7 @@ # This used to use the ImportHooksBaseTestCase to restore # the state of the import related information # in the sys module after each test. However, that restores - # *too much* information and breaks for the invocation of + # *too much* information and breaks for the invocation # of test_doctest. So we do our own thing and leave # sys.modules alone. # We also clear the linecache and zipimport cache diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -125,7 +125,7 @@ This class is the reference implementation of the Element interface. An element's length is its number of subelements. That means if you - you want to check if an element is truly empty, you should check BOTH + want to check if an element is truly empty, you should check BOTH its length AND its text attribute. The element tag, attribute names, and attribute values can be either diff --git a/Misc/HISTORY b/Misc/HISTORY --- a/Misc/HISTORY +++ b/Misc/HISTORY @@ -3972,7 +3972,7 @@ - Issue #12041: Make test_wait3 more robust. - Issue #11873: Change regex in test_compileall to fix occasional failures when - when the randomly generated temporary path happened to match the regex. + the randomly generated temporary path happened to match the regex. - Issue #11958: Fix FTP tests for IPv6, bind to "::1" instead of "localhost". Patch written by Charles-Francois Natali. @@ -14018,7 +14018,7 @@ - When method objects have an attribute that can be satisfied either by the function object or by the method object, the function object's attribute usually wins. Christian Tismer pointed out that - that this is really a mistake, because this only happens for special + this is really a mistake, because this only happens for special methods (like __reduce__) where the method object's version is really more appropriate than the function's attribute. So from now on, all method attributes will have precedence over function diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1362,7 +1362,7 @@ - Issue #15304: concurrent.futures.wait() can block forever even if Futures have completed. Patch by Glenn Langford. -- Issue #14455: plistlib: fix serializing integers integers in the range +- Issue #14455: plistlib: fix serializing integers in the range of an unsigned long long but outside of the range of signed long long for binary plist files. @@ -1373,7 +1373,7 @@ Patch mostly by Serhiy Storchaka. - Update the python.gif icon for the Idle classbrowser and pathbowser - from the old green snake to the new new blue and yellow snakes. + from the old green snake to the new blue and yellow snakes. - Issue #17721: Remove non-functional configuration dialog help button until we make it actually gives some help when clicked. Patch by Guilherme Sim?es. @@ -2654,7 +2654,7 @@ reside in the os module. - Issue #19205: Don't import the 're' module in site and sysconfig module to - to speed up interpreter start. + speed up interpreter start. - Issue #9548: Add a minimal "_bootlocale" module that is imported by the _io module instead of the full locale module. diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -104,7 +104,7 @@ Protected by TABLES_LOCK(). */ static size_t tracemalloc_peak_traced_memory = 0; -/* Hash table used as a set to to intern filenames: +/* Hash table used as a set to intern filenames: PyObject* => PyObject*. Protected by the GIL */ static _Py_hashtable_t *tracemalloc_filenames = NULL; diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -10289,7 +10289,7 @@ /* This code is used to ensure that the tables of configuration value names - * are in sorted order as required by conv_confname(), and also to build the + * are in sorted order as required by conv_confname(), and also to build * the exported dictionaries that are used to publish information about the * names available on the host platform. * diff --git a/Python/pystrtod.c b/Python/pystrtod.c --- a/Python/pystrtod.c +++ b/Python/pystrtod.c @@ -325,7 +325,7 @@ On overflow (e.g., when trying to convert '1e500' on an IEEE 754 machine), if overflow_exception is NULL then +-Py_HUGE_VAL is returned, and no Python - exception is raised. Otherwise, overflow_exception should point to a + exception is raised. Otherwise, overflow_exception should point to a Python exception, this exception will be raised, -1.0 will be returned, and *endptr will point just past the end of the converted value. diff --git a/Python/random.c b/Python/random.c --- a/Python/random.c +++ b/Python/random.c @@ -227,7 +227,7 @@ } /* Fill buffer with size pseudo-random bytes from the operating system random - number generator (RNG). It is suitable for for most cryptographic purposes + number generator (RNG). It is suitable for most cryptographic purposes except long living private keys for asymmetric encryption. Return 0 on success, raise an exception and return -1 on error. */ diff --git a/Python/thread.c b/Python/thread.c --- a/Python/thread.c +++ b/Python/thread.c @@ -221,7 +221,7 @@ goto Done; } /* Sanity check. These states should never happen but if - * they do we must abort. Otherwise we'll end up spinning in + * they do we must abort. Otherwise we'll end up spinning * in a tight loop with the lock held. A similar check is done * in pystate.c tstate_delete_common(). */ if (p == prev_p) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 17:59:23 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 16:59:23 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyOTc1?= =?utf-8?q?=3A_Close_block_at_right_place=2E?= Message-ID: <20141201165921.84275.40403@psf.io> https://hg.python.org/cpython/rev/15a1070a2c66 changeset: 93684:15a1070a2c66 branch: 3.4 parent: 93682:a4f3e78f68e4 user: Serhiy Storchaka date: Mon Dec 01 18:56:28 2014 +0200 summary: Issue #22975: Close block at right place. files: Objects/unicodeobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -9891,9 +9891,9 @@ Py_UCS4 * to_ = (Py_UCS4 *)((data)) + (start); \ for (; i_ < (length); ++i_, ++to_) *to_ = (value); \ break; \ + } \ default: assert(0); \ } \ - } \ } while (0) void -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 17:59:23 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 01 Dec 2014 16:59:23 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322975=3A_Close_block_at_right_place=2E?= Message-ID: <20141201165921.126782.2525@psf.io> https://hg.python.org/cpython/rev/e1afae23dfcd changeset: 93685:e1afae23dfcd parent: 93683:506d22434d63 parent: 93684:15a1070a2c66 user: Serhiy Storchaka date: Mon Dec 01 18:56:54 2014 +0200 summary: Issue #22975: Close block at right place. files: Objects/unicodeobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -9880,9 +9880,9 @@ Py_UCS4 * to_ = (Py_UCS4 *)((data)) + (start); \ for (; i_ < (length); ++i_, ++to_) *to_ = (value); \ break; \ + } \ default: assert(0); \ } \ - } \ } while (0) void -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 23:02:07 2014 From: python-checkins at python.org (terry.reedy) Date: Mon, 01 Dec 2014 22:02:07 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Add_NEWS_items?= =?utf-8?q?_for_Idle_patches_and_a_turtledemo_patch=2E?= Message-ID: <20141201220203.69783.38438@psf.io> https://hg.python.org/cpython/rev/de898ba9941d changeset: 93687:de898ba9941d branch: 3.4 parent: 93684:15a1070a2c66 user: Terry Jan Reedy date: Mon Dec 01 16:59:09 2014 -0500 summary: Add NEWS items for Idle patches and a turtledemo patch. files: Misc/NEWS | 24 +++++++++++++++++++++++- 1 files changed, 23 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -36,6 +36,8 @@ Library ------- +- Issue #21971: Update turtledemo doc and add module to the index. + - Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. Original patch by Martin Panter. @@ -90,7 +92,7 @@ - Issue #22776: Brought excluded code into the scope of a try block in SysLogHandler.emit(). - + - Issue #22665: Add missing get_terminal_size and SameFileError to shutil.__all__. @@ -158,6 +160,26 @@ - Issue #22448: Improve canceled timer handles cleanup to prevent unbound memory usage. Patch by Joshua Moore-Oliva. +IDLE +---- + +- Issue #3068: Add Idle extension configuration dialog to Options menu. + Changes are written to HOME/.idlerc/config-extensions.cfg. + Original patch by Tal Einat. + +- Issue #16233: A module browser (File : Class Browser, Alt+C) requires a + editor window with a filename. When Class Browser is requested otherwise, + from a shell, output window, or 'Untitled' editor, Idle no longer displays + an error box. It now pops up an Open Module box (Alt+M). If a valid name + is entered and a module is opened, a corresponding browser is also opened. + +- Issue #4832: Save As to type Python files automatically adds .py to the + name you enter (even if your system does not display it). Some systems + automatically add .txt when type is Text files. + +- Issue #21986: Code objects are not normally pickled by the pickle module. + To match this, they are no longer pickled when running under Idle. + Tests ----- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 23:02:10 2014 From: python-checkins at python.org (terry.reedy) Date: Mon, 01 Dec 2014 22:02:10 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Add_NEWS_items?= =?utf-8?q?_for_Idle_patches_already_in_2=2E7=2E9rc1=2E?= Message-ID: <20141201220203.55101.15006@psf.io> https://hg.python.org/cpython/rev/66a261969c57 changeset: 93686:66a261969c57 branch: 2.7 parent: 93681:cc744bf70cf6 user: Terry Jan Reedy date: Mon Dec 01 16:59:02 2014 -0500 summary: Add NEWS items for Idle patches already in 2.7.9rc1. files: Misc/NEWS | 19 ++++++++++++++++++- 1 files changed, 18 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -122,7 +122,7 @@ - Issue #22776: Brought excluded code into the scope of a try block in SysLogHandler.emit(). - + - Issue #17381: Fixed ranges handling in case-insensitive regular expressions. - Issue #19329: Optimized compiling charsets in regular expressions. @@ -317,6 +317,23 @@ IDLE ---- +- Issue #3068: Add Idle extension configuration dialog to Options menu. + Changes are written to HOME/.idlerc/config-extensions.cfg. + Original patch by Tal Einat. + +- Issue #16233: A module browser (File : Class Browser, Alt+C) requires a + editor window with a filename. When Class Browser is requested otherwise, + from a shell, output window, or 'Untitled' editor, Idle no longer displays + an error box. It now pops up an Open Module box (Alt+M). If a valid name + is entered and a module is opened, a corresponding browser is also opened. + +- Issue #4832: Save As to type Python files automatically adds .py to the + name you enter (even if your system does not display it). Some systems + automatically add .txt when type is Text files. + +- Issue #21986: Code objects are not normally pickled by the pickle module. + To match this, they are no longer pickled when running under Idle. + - Issue #22221: IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 23:13:41 2014 From: python-checkins at python.org (terry.reedy) Date: Mon, 01 Dec 2014 22:13:41 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <20141201220908.116314.35873@psf.io> https://hg.python.org/cpython/rev/a1aa0a4371eb changeset: 93688:a1aa0a4371eb parent: 93685:e1afae23dfcd parent: 93687:de898ba9941d user: Terry Jan Reedy date: Mon Dec 01 17:08:53 2014 -0500 summary: Merge with 3.4 files: Misc/NEWS | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -191,6 +191,8 @@ Library ------- +- Issue #21971: Update turtledemo doc and add module to the index. + - Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. Original patch by Martin Panter. @@ -1186,6 +1188,23 @@ IDLE ---- +- Issue #3068: Add Idle extension configuration dialog to Options menu. + Changes are written to HOME/.idlerc/config-extensions.cfg. + Original patch by Tal Einat. + +- Issue #16233: A module browser (File : Class Browser, Alt+C) requires a + editor window with a filename. When Class Browser is requested otherwise, + from a shell, output window, or 'Untitled' editor, Idle no longer displays + an error box. It now pops up an Open Module box (Alt+M). If a valid name + is entered and a module is opened, a corresponding browser is also opened. + +- Issue #4832: Save As to type Python files automatically adds .py to the + name you enter (even if your system does not display it). Some systems + automatically add .txt when type is Text files. + +- Issue #21986: Code objects are not normally pickled by the pickle module. + To match this, they are no longer pickled when running under Idle. + - Issue #17390: Adjust Editor window title; remove 'Python', move version to end. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 1 23:17:10 2014 From: python-checkins at python.org (terry.reedy) Date: Mon, 01 Dec 2014 22:17:10 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_spurious_header_in_?= =?utf-8?q?middle_of_Library_section=2E?= Message-ID: <20141201221655.126778.92337@psf.io> https://hg.python.org/cpython/rev/8eff0fb44013 changeset: 93689:8eff0fb44013 user: Terry Jan Reedy date: Mon Dec 01 17:16:43 2014 -0500 summary: Remove spurious header in middle of Library section. I suspect this was leftover from a merge conflict resolution. files: Misc/NEWS | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -399,9 +399,6 @@ - Issue #17442: InteractiveInterpreter now displays the full chained traceback in its showtraceback method, to match the built in interactive interpreter. -Tools/Demos ------------ - - Issue #22314: pydoc now works when the LINES environment variable is set. - Issue #10510: distutils register and upload methods now use HTML standards -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 00:17:49 2014 From: python-checkins at python.org (barry.warsaw) Date: Mon, 01 Dec 2014 23:17:49 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40Lg==?= Message-ID: <20141201231740.69783.48275@psf.io> https://hg.python.org/cpython/rev/633565b34b8b changeset: 93694:633565b34b8b parent: 93692:25113281d543 parent: 93693:8e18c049fb2f user: Barry Warsaw date: Mon Dec 01 18:16:12 2014 -0500 summary: Merge 3.4. files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 00:17:49 2014 From: python-checkins at python.org (barry.warsaw) Date: Mon, 01 Dec 2014 23:17:49 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_-_Issue_=2322966=3A_Fix_=5F=5Fpycache=5F=5F_pyc_file_nam?= =?utf-8?q?e_clobber_when_pyc=5Fcompile_is?= Message-ID: <20141201231739.55117.89046@psf.io> https://hg.python.org/cpython/rev/25113281d543 changeset: 93692:25113281d543 parent: 93688:a1aa0a4371eb parent: 93691:3b3ba38d503a user: Barry Warsaw date: Mon Dec 01 17:52:43 2014 -0500 summary: - Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is asked to compile a source file containing multiple dots in the source file name. files: Lib/importlib/_bootstrap.py | 4 +- Lib/test/test_py_compile.py | 17 + Misc/NEWS | 4 + Python/importlib.h | 7013 +++++++++++----------- 4 files changed, 3530 insertions(+), 3508 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -454,11 +454,11 @@ else: suffixes = OPTIMIZED_BYTECODE_SUFFIXES head, tail = _path_split(path) - base_filename, sep, _ = tail.partition('.') + base, sep, rest = tail.rpartition('.') tag = sys.implementation.cache_tag if tag is None: raise NotImplementedError('sys.implementation.cache_tag is None') - filename = ''.join([base_filename, sep, tag, suffixes[0]]) + filename = ''.join([(base if base else rest), sep, tag, suffixes[0]]) return _path_join(head, _PYCACHE, filename) diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -3,6 +3,7 @@ import py_compile import shutil import stat +import sys import tempfile import unittest @@ -99,5 +100,21 @@ self.assertFalse(os.path.exists( importlib.util.cache_from_source(bad_coding))) + def test_double_dot_no_clobber(self): + # http://bugs.python.org/issue22966 + # py_compile foo.bar.py -> __pycache__/foo.cpython-34.pyc + weird_path = os.path.join(self.directory, 'foo.bar.py') + cache_path = importlib.util.cache_from_source(weird_path) + pyc_path = weird_path + 'c' + self.assertEqual( + '/'.join(cache_path.split('/')[-2:]), + '__pycache__/foo.bar.{}.pyc'.format(sys.implementation.cache_tag)) + with open(weird_path, 'w') as file: + file.write('x = 123\n') + py_compile.compile(weird_path) + self.assertTrue(os.path.exists(cache_path)) + self.assertFalse(os.path.exists(pyc_path)) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -191,6 +191,10 @@ Library ------- +- Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is + asked to compile a source file containing multiple dots in the source file + name. + - Issue #21971: Update turtledemo doc and add module to the index. - Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 00:17:49 2014 From: python-checkins at python.org (barry.warsaw) Date: Mon, 01 Dec 2014 23:17:49 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogLSBJc3N1ZSAjMjI5?= =?utf-8?q?66=3A_Fix_=5F=5Fpycache=5F=5F_pyc_file_name_clobber_when_pyc=5F?= =?utf-8?q?compile_is?= Message-ID: <20141201231739.116310.85158@psf.io> https://hg.python.org/cpython/rev/269bf37a57a1 changeset: 93690:269bf37a57a1 branch: 3.4 parent: 93679:ba72da4883eb user: Barry Warsaw date: Mon Dec 01 17:10:10 2014 -0500 summary: - Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is asked to compile a source file containing multiple dots in the source file name. files: Lib/importlib/_bootstrap.py | 4 +- Lib/test/test_py_compile.py | 16 + Misc/NEWS | 4 + Python/importlib.h | 7255 +++++++++++----------- 4 files changed, 3650 insertions(+), 3629 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -453,11 +453,11 @@ else: suffixes = OPTIMIZED_BYTECODE_SUFFIXES head, tail = _path_split(path) - base_filename, sep, _ = tail.partition('.') + base, sep, rest = tail.rpartition('.') tag = sys.implementation.cache_tag if tag is None: raise NotImplementedError('sys.implementation.cache_tag is None') - filename = ''.join([base_filename, sep, tag, suffixes[0]]) + filename = ''.join([(base if base else rest), sep, tag, suffixes[0]]) return _path_join(head, _PYCACHE, filename) diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -99,5 +99,21 @@ self.assertFalse(os.path.exists( importlib.util.cache_from_source(bad_coding))) + def test_double_dot_no_clobber(self): + # http://bugs.python.org/issue22966 + # py_compile foo.bar.py -> __pycache__/foo.cpython-34.pyc + weird_path = os.path.join(self.directory, 'foo.bar.py') + cache_path = importlib.util.cache_from_source(weird_path) + pyc_path = weird_path + 'c' + self.assertEqual( + '/'.join(cache_path.split('/')[-2:]), + '__pycache__/foo.bar.cpython-34.pyc') + with open(weird_path, 'w') as file: + file.write('x = 123\n') + py_compile.compile(weird_path) + self.assertTrue(os.path.exists(cache_path)) + self.assertFalse(os.path.exists(pyc_path)) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -36,6 +36,10 @@ Library ------- +- Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is + asked to compile a source file containing multiple dots in the source file + name. + - Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. Original patch by Martin Panter. diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 00:17:54 2014 From: python-checkins at python.org (barry.warsaw) Date: Mon, 01 Dec 2014 23:17:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?b?KTogVHJ1bmsgbWVyZ2Uu?= Message-ID: <20141201231740.55109.17732@psf.io> https://hg.python.org/cpython/rev/823093db83eb changeset: 93695:823093db83eb parent: 93694:633565b34b8b parent: 93689:8eff0fb44013 user: Barry Warsaw date: Mon Dec 01 18:17:29 2014 -0500 summary: Trunk merge. files: Misc/NEWS | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -403,9 +403,6 @@ - Issue #17442: InteractiveInterpreter now displays the full chained traceback in its showtraceback method, to match the built in interactive interpreter. -Tools/Demos ------------ - - Issue #22314: pydoc now works when the LINES environment variable is set. - Issue #10510: distutils register and upload methods now use HTML standards -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 00:17:54 2014 From: python-checkins at python.org (barry.warsaw) Date: Mon, 01 Dec 2014 23:17:54 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Make_this_chan?= =?utf-8?q?ge_a_little_more_robust_for_up-merging=2E?= Message-ID: <20141201231739.69803.85466@psf.io> https://hg.python.org/cpython/rev/8e18c049fb2f changeset: 93693:8e18c049fb2f branch: 3.4 parent: 93691:3b3ba38d503a user: Barry Warsaw date: Mon Dec 01 18:15:26 2014 -0500 summary: Make this change a little more robust for up-merging. files: Lib/test/test_py_compile.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -3,6 +3,7 @@ import py_compile import shutil import stat +import sys import tempfile import unittest @@ -107,7 +108,7 @@ pyc_path = weird_path + 'c' self.assertEqual( '/'.join(cache_path.split('/')[-2:]), - '__pycache__/foo.bar.cpython-34.pyc') + '__pycache__/foo.bar.{}.pyc'.format(sys.implementation.cache_tag)) with open(weird_path, 'w') as file: file.write('x = 123\n') py_compile.compile(weird_path) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 00:17:54 2014 From: python-checkins at python.org (barry.warsaw) Date: Mon, 01 Dec 2014 23:17:54 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy40IC0+IDMuNCk6?= =?utf-8?q?_-_Issue_=2322966=3A_Fix_=5F=5Fpycache=5F=5F_pyc_file_name_clob?= =?utf-8?q?ber_when_pyc=5Fcompile_is?= Message-ID: <20141201231739.84305.83245@psf.io> https://hg.python.org/cpython/rev/3b3ba38d503a changeset: 93691:3b3ba38d503a branch: 3.4 parent: 93687:de898ba9941d parent: 93690:269bf37a57a1 user: Barry Warsaw date: Mon Dec 01 17:23:55 2014 -0500 summary: - Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is asked to compile a source file containing multiple dots in the source file name. files: Lib/importlib/_bootstrap.py | 4 +- Lib/test/test_py_compile.py | 16 + Misc/NEWS | 4 + Python/importlib.h | 7255 +++++++++++----------- 4 files changed, 3650 insertions(+), 3629 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -453,11 +453,11 @@ else: suffixes = OPTIMIZED_BYTECODE_SUFFIXES head, tail = _path_split(path) - base_filename, sep, _ = tail.partition('.') + base, sep, rest = tail.rpartition('.') tag = sys.implementation.cache_tag if tag is None: raise NotImplementedError('sys.implementation.cache_tag is None') - filename = ''.join([base_filename, sep, tag, suffixes[0]]) + filename = ''.join([(base if base else rest), sep, tag, suffixes[0]]) return _path_join(head, _PYCACHE, filename) diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -99,5 +99,21 @@ self.assertFalse(os.path.exists( importlib.util.cache_from_source(bad_coding))) + def test_double_dot_no_clobber(self): + # http://bugs.python.org/issue22966 + # py_compile foo.bar.py -> __pycache__/foo.cpython-34.pyc + weird_path = os.path.join(self.directory, 'foo.bar.py') + cache_path = importlib.util.cache_from_source(weird_path) + pyc_path = weird_path + 'c' + self.assertEqual( + '/'.join(cache_path.split('/')[-2:]), + '__pycache__/foo.bar.cpython-34.pyc') + with open(weird_path, 'w') as file: + file.write('x = 123\n') + py_compile.compile(weird_path) + self.assertTrue(os.path.exists(cache_path)) + self.assertFalse(os.path.exists(pyc_path)) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -36,6 +36,10 @@ Library ------- +- Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is + asked to compile a source file containing multiple dots in the source file + name. + - Issue #21971: Update turtledemo doc and add module to the index. - Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 00:20:28 2014 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 01 Dec 2014 23:20:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_uninitialized_variable?= =?utf-8?q?_after_=2322676=2E?= Message-ID: <20141201232013.116308.38711@psf.io> https://hg.python.org/cpython/rev/3e3bec66409c changeset: 93696:3e3bec66409c user: Antoine Pitrou date: Tue Dec 02 00:20:03 2014 +0100 summary: Fix uninitialized variable after #22676. files: Lib/test/pickletester.py | 21 +++++++++++++++++++++ Modules/_pickle.c | 24 +++++++++++++++++------- 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -1636,6 +1636,27 @@ unpickled = self.loads(self.dumps(method, proto)) self.assertEqual(method(*args), unpickled(*args)) + def test_local_lookup_error(self): + # Test that whichmodule() errors out cleanly when looking up + # an assumed globally-reachable object fails. + def f(): + pass + # Since the function is local, lookup will fail + for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): + with self.assertRaises((AttributeError, pickle.PicklingError)): + pickletools.dis(self.dumps(f, proto)) + # Same without a __module__ attribute (exercises a different path + # in _pickle.c). + del f.__module__ + for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): + with self.assertRaises((AttributeError, pickle.PicklingError)): + pickletools.dis(self.dumps(f, proto)) + # Yet a different path. + f.__name__ = f.__qualname__ + for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): + with self.assertRaises((AttributeError, pickle.PicklingError)): + pickletools.dis(self.dumps(f, proto)) + class BigmemPickleTests(unittest.TestCase): diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -1547,10 +1547,16 @@ n = PyList_GET_SIZE(dotted_path); assert(n >= 1); if (!allow_qualname && n > 1) { - PyErr_Format(PyExc_AttributeError, - "Can't get qualified attribute %R on %R;" - "use protocols >= 4 to enable support", - name, obj); + if (obj == NULL) + PyErr_Format(PyExc_AttributeError, + "Can't pickle qualified object %R; " + "use protocols >= 4 to enable support", + name); + else + PyErr_Format(PyExc_AttributeError, + "Can't pickle qualified attribute %R on %R; " + "use protocols >= 4 to enable support", + name, obj); Py_DECREF(dotted_path); return NULL; } @@ -1562,8 +1568,12 @@ assert(PyBool_Check(result)); Py_DECREF(result); if (is_equal) { - PyErr_Format(PyExc_AttributeError, - "Can't get local attribute %R on %R", name, obj); + if (obj == NULL) + PyErr_Format(PyExc_AttributeError, + "Can't pickle local object %R", name); + else + PyErr_Format(PyExc_AttributeError, + "Can't pickle local attribute %R on %R", name, obj); Py_DECREF(dotted_path); return NULL; } @@ -1653,7 +1663,7 @@ return NULL; } - dotted_path = get_dotted_path(module, global_name, allow_qualname); + dotted_path = get_dotted_path(NULL, global_name, allow_qualname); if (dotted_path == NULL) return NULL; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 01:33:21 2014 From: python-checkins at python.org (georg.brandl) Date: Tue, 02 Dec 2014 00:33:21 +0000 Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_Chris_Angelico=2E?= Message-ID: <20141202003302.55115.83503@psf.io> https://hg.python.org/devguide/rev/9b574ea00cde changeset: 725:9b574ea00cde user: Georg Brandl date: Tue Dec 02 01:32:55 2014 +0100 summary: Add Chris Angelico. files: developers.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/developers.rst b/developers.rst --- a/developers.rst +++ b/developers.rst @@ -25,6 +25,9 @@ Permissions History ------------------- +- Chris Angelico was given push privileges on December 1, 2014 by GFB, + as a new PEP editor. + - Santoso Wijaya was given push privileges on October 29, 2014 by GFB, at the request of Frank Wierzbicki, for Jython development. -- Repository URL: https://hg.python.org/devguide From python-checkins at python.org Tue Dec 2 02:05:45 2014 From: python-checkins at python.org (chris.angelico) Date: Tue, 02 Dec 2014 01:05:45 +0000 Subject: [Python-checkins] =?utf-8?q?test=3A_Record_the_canonical_and_orig?= =?utf-8?q?inal_Lenaledian_paragraph_from_the_original_CCC?= Message-ID: <20141202010536.84291.72933@psf.io> https://hg.python.org/test/rev/62178dc71971 changeset: 219:62178dc71971 user: Chris Angelico date: Tue Dec 02 12:00:33 2014 +1100 summary: Record the canonical and original Lenaledian paragraph from the original CCC Makes for a good test. Also, my Camembert Canyon Campaign is finished now, so there's no reason not to use this as a basic test of push privileges, which (after all) is what this repo exists for. With tests like these, of course, actual content is 100% optional, but hey :) nothing wrong with a quick demo of what can be done while testing what can be done. So check out the column over here. Have fun. ^ files: Lenalede | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-) diff --git a/Lenalede b/Lenalede new file mode 100644 --- /dev/null +++ b/Lenalede @@ -0,0 +1,13 @@ +.:..............:..........................:........................:........... +Ignoble or ranked, male or female, full-blood or half-blood, all must unite! +Cast out our tendentious lifetime-appointed ruling class and loose the citizens, +as our Powerhouses Charter advocates. Why, friends, can this city's researchers +attach unscientific "statistics" to twofolds to maintain a semblance of support? +No judges should boast honesty beyond object, much less infallability; however +unfair the "judgements" are with Lenalede's Lord Chief Justice now, it is plain +a clear, honest assessment has to be made. Colinette vows reform. If among Lords +nobody sets constitutionality a priority, can we demand "Resignations And Role +Obituaries" to finally permit this city to maintain the substructures that make +Lenalede your city of free will, giving zones to citizens meriting? +.........:.................:..............................:..................... + -- Repository URL: https://hg.python.org/test From python-checkins at python.org Tue Dec 2 08:35:19 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 02 Dec 2014 07:35:19 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Moved_Misc/NEWS_entry_to_r?= =?utf-8?q?ight_section=2E?= Message-ID: <20141202073515.69797.50380@psf.io> https://hg.python.org/cpython/rev/15b35fc21995 changeset: 93697:15b35fc21995 user: Serhiy Storchaka date: Tue Dec 02 09:13:10 2014 +0200 summary: Moved Misc/NEWS entry to right section. files: Misc/NEWS | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -403,8 +403,6 @@ - Issue #17442: InteractiveInterpreter now displays the full chained traceback in its showtraceback method, to match the built in interactive interpreter. -- Issue #22314: pydoc now works when the LINES environment variable is set. - - Issue #10510: distutils register and upload methods now use HTML standards compliant CRLF line endings. @@ -1469,6 +1467,8 @@ Tools/Demos ----------- +- Issue #22314: pydoc now works when the LINES environment variable is set. + - Issue #22615: Argument Clinic now supports the "type" argument for the int converter. This permits using the int converter with enums and typedefs. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 08:35:19 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 02 Dec 2014 07:35:19 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320335=3A_bytes_constructor_now_raises_TypeError?= =?utf-8?q?_when_encoding_or_errors?= Message-ID: <20141202073515.126772.25781@psf.io> https://hg.python.org/cpython/rev/8d6b27837c69 changeset: 93699:8d6b27837c69 parent: 93697:15b35fc21995 parent: 93698:8badbd65840e user: Serhiy Storchaka date: Tue Dec 02 09:26:14 2014 +0200 summary: Issue #20335: bytes constructor now raises TypeError when encoding or errors is specified with non-string argument. Based on patch by Renaud Blanch. files: Lib/test/test_bytes.py | 8 ++++++++ Misc/NEWS | 3 +++ Objects/bytesobject.c | 14 +++++++------- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -100,6 +100,14 @@ self.assertRaises(TypeError, self.type2test, [0.0]) self.assertRaises(TypeError, self.type2test, [None]) self.assertRaises(TypeError, self.type2test, [C()]) + self.assertRaises(TypeError, self.type2test, 0, 'ascii') + self.assertRaises(TypeError, self.type2test, b'', 'ascii') + self.assertRaises(TypeError, self.type2test, 0, errors='ignore') + self.assertRaises(TypeError, self.type2test, b'', errors='ignore') + self.assertRaises(TypeError, self.type2test, '') + self.assertRaises(TypeError, self.type2test, '', errors='ignore') + self.assertRaises(TypeError, self.type2test, '', b'ascii') + self.assertRaises(TypeError, self.type2test, '', 'ascii', b'ignore') def test_constructor_value_errors(self): self.assertRaises(ValueError, self.type2test, [-1]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #20335: bytes constructor now raises TypeError when encoding or errors + is specified with non-string argument. Based on patch by Renaud Blanch. + - Issue #22834: If the current working directory ends up being set to a non-existent directory then import will no longer raise FileNotFoundError. diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -3039,6 +3039,13 @@ return new; } + /* If it's not unicode, there can't be encoding or errors */ + if (encoding != NULL || errors != NULL) { + PyErr_SetString(PyExc_TypeError, + "encoding or errors without a string argument"); + return NULL; + } + /* We'd like to call PyObject_Bytes here, but we need to check for an integer argument before deferring to PyBytes_FromObject, something PyObject_Bytes doesn't do. */ @@ -3078,13 +3085,6 @@ return new; } - /* If it's not unicode, there can't be encoding or errors */ - if (encoding != NULL || errors != NULL) { - PyErr_SetString(PyExc_TypeError, - "encoding or errors without a string argument"); - return NULL; - } - return PyBytes_FromObject(x); } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 08:35:19 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 02 Dec 2014 07:35:19 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwMzM1?= =?utf-8?q?=3A_bytes_constructor_now_raises_TypeError_when_encoding_or_err?= =?utf-8?q?ors?= Message-ID: <20141202073515.55107.52346@psf.io> https://hg.python.org/cpython/rev/8badbd65840e changeset: 93698:8badbd65840e branch: 3.4 parent: 93693:8e18c049fb2f user: Serhiy Storchaka date: Tue Dec 02 09:24:06 2014 +0200 summary: Issue #20335: bytes constructor now raises TypeError when encoding or errors is specified with non-string argument. Based on patch by Renaud Blanch. files: Lib/test/test_bytes.py | 8 ++++++++ Misc/NEWS | 3 +++ Objects/bytesobject.c | 14 +++++++------- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -100,6 +100,14 @@ self.assertRaises(TypeError, self.type2test, [0.0]) self.assertRaises(TypeError, self.type2test, [None]) self.assertRaises(TypeError, self.type2test, [C()]) + self.assertRaises(TypeError, self.type2test, 0, 'ascii') + self.assertRaises(TypeError, self.type2test, b'', 'ascii') + self.assertRaises(TypeError, self.type2test, 0, errors='ignore') + self.assertRaises(TypeError, self.type2test, b'', errors='ignore') + self.assertRaises(TypeError, self.type2test, '') + self.assertRaises(TypeError, self.type2test, '', errors='ignore') + self.assertRaises(TypeError, self.type2test, '', b'ascii') + self.assertRaises(TypeError, self.type2test, '', 'ascii', b'ignore') def test_constructor_value_errors(self): self.assertRaises(ValueError, self.type2test, [-1]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -11,6 +11,9 @@ Core and Builtins ----------------- +- Issue #20335: bytes constructor now raises TypeError when encoding or errors + is specified with non-string argument. Based on patch by Renaud Blanch. + - Issue #22335: Fix crash when trying to enlarge a bytearray to 0x7fffffff bytes on a 32-bit platform. diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -2503,6 +2503,13 @@ return new; } + /* If it's not unicode, there can't be encoding or errors */ + if (encoding != NULL || errors != NULL) { + PyErr_SetString(PyExc_TypeError, + "encoding or errors without a string argument"); + return NULL; + } + /* We'd like to call PyObject_Bytes here, but we need to check for an integer argument before deferring to PyBytes_FromObject, something PyObject_Bytes doesn't do. */ @@ -2544,13 +2551,6 @@ return new; } - /* If it's not unicode, there can't be encoding or errors */ - if (encoding != NULL || errors != NULL) { - PyErr_SetString(PyExc_TypeError, - "encoding or errors without a string argument"); - return NULL; - } - return PyBytes_FromObject(x); } -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Tue Dec 2 10:33:19 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 02 Dec 2014 10:33:19 +0100 Subject: [Python-checkins] Daily reference leaks (3e3bec66409c): sum=3 Message-ID: results for 3e3bec66409c on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogWWWqCn', '-x'] From python-checkins at python.org Tue Dec 2 17:31:52 2014 From: python-checkins at python.org (barry.warsaw) Date: Tue, 02 Dec 2014 16:31:52 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy40Lg==?= Message-ID: <20141202163122.69785.91361@psf.io> https://hg.python.org/cpython/rev/0744ceb5c0ed changeset: 93701:0744ceb5c0ed parent: 93699:8d6b27837c69 parent: 93700:4e9c495235d9 user: Barry Warsaw date: Tue Dec 02 11:31:10 2014 -0500 summary: Merge 3.4. files: Lib/test/test_py_compile.py | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -106,9 +106,13 @@ weird_path = os.path.join(self.directory, 'foo.bar.py') cache_path = importlib.util.cache_from_source(weird_path) pyc_path = weird_path + 'c' + head, tail = os.path.split(cache_path) + penultimate_tail = os.path.basename(head) self.assertEqual( - '/'.join(cache_path.split('/')[-2:]), - '__pycache__/foo.bar.{}.pyc'.format(sys.implementation.cache_tag)) + os.path.join(penultimate_tail, tail), + os.path.join( + '__pycache__', + 'foo.bar.{}.pyc'.format(sys.implementation.cache_tag))) with open(weird_path, 'w') as file: file.write('x = 123\n') py_compile.compile(weird_path) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 17:31:52 2014 From: python-checkins at python.org (barry.warsaw) Date: Tue, 02 Dec 2014 16:31:52 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_the_test_t?= =?utf-8?q?o_use_an_os=2Esep_agnostic_test=2E__Hopefully_this_will_fix_the?= Message-ID: <20141202163121.116324.49074@psf.io> https://hg.python.org/cpython/rev/4e9c495235d9 changeset: 93700:4e9c495235d9 branch: 3.4 parent: 93698:8badbd65840e user: Barry Warsaw date: Tue Dec 02 11:30:43 2014 -0500 summary: Fix the test to use an os.sep agnostic test. Hopefully this will fix the Windows buildbots. Found by Jeremy Kloth. files: Lib/test/test_py_compile.py | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -106,9 +106,13 @@ weird_path = os.path.join(self.directory, 'foo.bar.py') cache_path = importlib.util.cache_from_source(weird_path) pyc_path = weird_path + 'c' + head, tail = os.path.split(cache_path) + penultimate_tail = os.path.basename(head) self.assertEqual( - '/'.join(cache_path.split('/')[-2:]), - '__pycache__/foo.bar.{}.pyc'.format(sys.implementation.cache_tag)) + os.path.join(penultimate_tail, tail), + os.path.join( + '__pycache__', + 'foo.bar.{}.pyc'.format(sys.implementation.cache_tag))) with open(weird_path, 'w') as file: file.write('x = 123\n') py_compile.compile(weird_path) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 17:53:15 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 02 Dec 2014 16:53:15 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?= Message-ID: <20141202165304.55103.76477@psf.io> https://hg.python.org/cpython/rev/c92c4cb5d5f7 changeset: 93703:c92c4cb5d5f7 parent: 93701:0744ceb5c0ed parent: 93702:a4b58e779a16 user: Victor Stinner date: Tue Dec 02 17:52:57 2014 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-task.rst | 11 +++-------- 1 files changed, 3 insertions(+), 8 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -337,14 +337,9 @@ finally: loop.close() -In this example, the future is responsible to display the result and to stop -the loop. - -.. note:: - The "slow_operation" coroutine object is only executed when the event loop - starts running, so it is possible to add a "done callback" to the future - after creating the task scheduling the coroutine object. - +In this example, the future is used to link ``slow_operation()`` to +``got_result()``: when ``slow_operation()`` is done, ``got_result()`` is called +with the result. Task -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 17:53:15 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 02 Dec 2014 16:53:15 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2UgIzIyNDcz?= =?utf-8?q?=3A_asyncio_doc=3A_rephrase_Future_with_run=5Fforever=28=29_exa?= =?utf-8?q?mple?= Message-ID: <20141202165304.116310.80174@psf.io> https://hg.python.org/cpython/rev/a4b58e779a16 changeset: 93702:a4b58e779a16 branch: 3.4 parent: 93700:4e9c495235d9 user: Victor Stinner date: Tue Dec 02 17:52:45 2014 +0100 summary: Close #22473: asyncio doc: rephrase Future with run_forever() example files: Doc/library/asyncio-task.rst | 11 +++-------- 1 files changed, 3 insertions(+), 8 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -337,14 +337,9 @@ finally: loop.close() -In this example, the future is responsible to display the result and to stop -the loop. - -.. note:: - The "slow_operation" coroutine object is only executed when the event loop - starts running, so it is possible to add a "done callback" to the future - after creating the task scheduling the coroutine object. - +In this example, the future is used to link ``slow_operation()`` to +``got_result()``: when ``slow_operation()`` is done, ``got_result()`` is called +with the result. Task -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 18:01:52 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 02 Dec 2014 17:01:52 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMjQ3?= =?utf-8?q?5=3A_asyncio_doc=2C_fix_Task=2Eget=5Fstack=28=29_doc?= Message-ID: <20141202170135.55107.1582@psf.io> https://hg.python.org/cpython/rev/4b6b03c1f4ff changeset: 93704:4b6b03c1f4ff branch: 3.4 parent: 93702:a4b58e779a16 user: Victor Stinner date: Tue Dec 02 17:57:04 2014 +0100 summary: Closes #22475: asyncio doc, fix Task.get_stack() doc files: Doc/library/asyncio-task.rst | 2 +- Lib/asyncio/tasks.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -413,7 +413,7 @@ Return the list of stack frames for this task's coroutine. - If the coroutine is active, this returns the stack where it is suspended. + If the coroutine is not done, this returns the stack where it is suspended. If the coroutine has completed successfully or was cancelled, this returns an empty list. If the coroutine was terminated by an exception, this returns the list of traceback frames. diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -109,7 +109,7 @@ def get_stack(self, *, limit=None): """Return the list of stack frames for this task's coroutine. - If the coroutine is active, this returns the stack where it is + If the coroutine is not done, this returns the stack where it is suspended. If the coroutine has completed successfully or was cancelled, this returns an empty list. If the coroutine was terminated by an exception, this returns the list of traceback -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 18:01:52 2014 From: python-checkins at python.org (victor.stinner) Date: Tue, 02 Dec 2014 17:01:52 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?= Message-ID: <20141202170135.55115.99492@psf.io> https://hg.python.org/cpython/rev/3dbed7a37730 changeset: 93705:3dbed7a37730 parent: 93703:c92c4cb5d5f7 parent: 93704:4b6b03c1f4ff user: Victor Stinner date: Tue Dec 02 18:01:07 2014 +0100 summary: Merge 3.4 (asyncio doc) files: Doc/library/asyncio-task.rst | 2 +- Lib/asyncio/tasks.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -413,7 +413,7 @@ Return the list of stack frames for this task's coroutine. - If the coroutine is active, this returns the stack where it is suspended. + If the coroutine is not done, this returns the stack where it is suspended. If the coroutine has completed successfully or was cancelled, this returns an empty list. If the coroutine was terminated by an exception, this returns the list of traceback frames. diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -109,7 +109,7 @@ def get_stack(self, *, limit=None): """Return the list of stack frames for this task's coroutine. - If the coroutine is active, this returns the stack where it is + If the coroutine is not done, this returns the stack where it is suspended. If the coroutine has completed successfully or was cancelled, this returns an empty list. If the coroutine was terminated by an exception, this returns the list of traceback -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 2 22:41:58 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 02 Dec 2014 21:41:58 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2317401=3A_Output_t?= =?utf-8?q?he_closefd_attribute_as_boolean=2E?= Message-ID: <20141202214149.90399.81606@psf.io> https://hg.python.org/cpython/rev/fd80195b920f changeset: 93706:fd80195b920f user: Serhiy Storchaka date: Tue Dec 02 23:39:56 2014 +0200 summary: Issue #17401: Output the closefd attribute as boolean. files: Lib/test/test_fileio.py | 18 ++++++++++++++---- Modules/_io/fileio.c | 8 ++++---- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py --- a/Lib/test/test_fileio.py +++ b/Lib/test/test_fileio.py @@ -113,15 +113,25 @@ def testRepr(self): self.assertEqual( - repr(self.f), "<_io.FileIO name=%r mode=%r closefd='%d'>" - % (self.f.name, self.f.mode, self.f.closefd)) + repr(self.f), "<_io.FileIO name=%r mode=%r closefd=True>" + % (self.f.name, self.f.mode)) del self.f.name self.assertEqual( - repr(self.f), "<_io.FileIO fd=%r mode=%r closefd='%d'>" - % (self.f.fileno(), self.f.mode, self.f.closefd)) + repr(self.f), "<_io.FileIO fd=%r mode=%r closefd=True>" + % (self.f.fileno(), self.f.mode)) self.f.close() self.assertEqual(repr(self.f), "<_io.FileIO [closed]>") + def testReprNoCloseFD(self): + fd = os.open(TESTFN, os.O_RDONLY) + try: + with _FileIO(fd, 'r', closefd=False) as f: + self.assertEqual(repr(f), + "<_io.FileIO name=%r mode=%r closefd=False>" + % (f.name, f.mode)) + finally: + os.close(fd) + def testErrors(self): f = self.f self.assertTrue(not f.isatty()) diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -1055,13 +1055,13 @@ else return NULL; res = PyUnicode_FromFormat( - "<_io.FileIO fd=%d mode='%s' closefd='%d'>", - self->fd, mode_string(self), self->closefd); + "<_io.FileIO fd=%d mode='%s' closefd=%s>", + self->fd, mode_string(self), self->closefd ? "True" : "False"); } else { res = PyUnicode_FromFormat( - "<_io.FileIO name=%R mode='%s' closefd='%d'>", - nameobj, mode_string(self), self->closefd); + "<_io.FileIO name=%R mode='%s' closefd=%s>", + nameobj, mode_string(self), self->closefd ? "True" : "False"); Py_DECREF(nameobj); } return res; -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 3 00:45:50 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 02 Dec 2014 23:45:50 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_nuke_ancient_s?= =?utf-8?q?cript_with_dubious_license_=28closes_=2312987=29?= Message-ID: <20141202234548.4189.48699@psf.io> https://hg.python.org/cpython/rev/54af09408795 changeset: 93707:54af09408795 branch: 2.7 parent: 93667:8a3807e15a1f user: Benjamin Peterson date: Tue Dec 02 18:45:23 2014 -0500 summary: nuke ancient script with dubious license (closes #12987) files: Demo/scripts/newslist.doc | 59 ---- Demo/scripts/newslist.py | 362 -------------------------- 2 files changed, 0 insertions(+), 421 deletions(-) diff --git a/Demo/scripts/newslist.doc b/Demo/scripts/newslist.doc deleted file mode 100644 --- a/Demo/scripts/newslist.doc +++ /dev/null @@ -1,59 +0,0 @@ - NEWSLIST - ======== - A program to assist HTTP browsing of newsgroups - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -WWW browsers such as NCSA Mosaic allow the user to read newsgroup -articles by specifying the group name in a URL eg 'news:comp.answers'. - -To browse through many groups, though, (and there are several thousand -of them) you really need a page or pages containing links to all the -groups. There are some good ones out there, for example, - - http://info.cern.ch/hypertext/DataSources/News/Groups/Overview.html - -is the standard one at CERN, but it only shows the groups available there, -which may be rather different from those available on your machine. - -Newslist is a program which creates a hierarchy of pages for you based -on the groups available from YOUR server. It is written in python - a -splendid interpreted object-oriented language which I suggest you get -right now from the directory /pub/python at ftp.cwi.nl, if you haven't -already got it. - -You should be able to see some sample output by looking at: - http://pelican.cl.cam.ac.uk/newspage/root.html - -Descriptions of newsgroups can be added from a file with one group -per line. eg: - - alt.foo Articles about foo - comp.bar Programming in 'bar' and related languages - -A suitable list detailing most groups can be found at ftp.uu.net in -/uunet-info/newsgroups.gz. - -Make sure you read the information at the beginning of the program source and -configure the variables before running. - -In addition to python, you need: - - An NNTP-based news feed. - A directory in which to put the pages. - -The programming is not very beautiful, but it works! It comes with no -warranty, express or implied, but with the hope that some others may -find it useful. - -Comments, improvements & suggestions welcomed. -Quentin Stafford-Fraser - - ---------------------------------------------------------------------- - Quentin Stafford-Fraser - http://pelican.cl.cam.ac.uk/people/qs101/me.html - - Cambridge University Computer Lab Rank Xerox Cambridge EuroPARC - qs101 at cl.cam.ac.uk fraser at europarc.xerox.com - Tel: +44 223 334411 Tel: +44 223 341521 - Fax: +44 223 334679 Fax: +44 223 341510 - ---------------------------------------------------------------------- diff --git a/Demo/scripts/newslist.py b/Demo/scripts/newslist.py deleted file mode 100755 --- a/Demo/scripts/newslist.py +++ /dev/null @@ -1,362 +0,0 @@ -#! /usr/bin/env python -####################################################################### -# Newslist $Revision$ -# -# Syntax: -# newslist [ -a ] -# -# This is a program to create a directory full of HTML pages -# which between them contain links to all the newsgroups available -# on your server. -# -# The -a option causes a complete list of all groups to be read from -# the server rather than just the ones which have appeared since last -# execution. This recreates the local list from scratch. Use this on -# the first invocation of the program, and from time to time thereafter. -# When new groups are first created they may appear on your server as -# empty groups. By default, empty groups are ignored by the -a option. -# However, these new groups will not be created again, and so will not -# appear in the server's list of 'new groups' at a later date. Hence it -# won't appear until you do a '-a' after some articles have appeared. -# -# I should really keep a list of ignored empty groups and re-check them -# for articles on every run, but I haven't got around to it yet. -# -# This assumes an NNTP news feed. -# -# Feel free to copy, distribute and modify this code for -# non-commercial use. If you make any useful modifications, let me -# know! -# -# (c) Quentin Stafford-Fraser 1994 -# fraser at europarc.xerox.com qs101 at cl.cam.ac.uk -# # -####################################################################### -import sys, nntplib, marshal, time, os - -####################################################################### -# Check these variables before running! # - -# Top directory. -# Filenames which don't start with / are taken as being relative to this. -topdir = os.path.expanduser('~/newspage') - -# The name of your NNTP host -# eg. -# newshost = 'nntp-serv.cl.cam.ac.uk' -# or use following to get the name from the NNTPSERVER environment -# variable: -# newshost = os.environ['NNTPSERVER'] -newshost = 'news.example.com' - -# The filename for a local cache of the newsgroup list -treefile = 'grouptree' - -# The filename for descriptions of newsgroups -# I found a suitable one at ftp.uu.net in /uunet-info/newgroups.gz -# You can set this to '' if you don't wish to use one. -descfile = 'newsgroups' - -# The directory in which HTML pages should be created -# eg. -# pagedir = '/usr/local/lib/html/newspage' -# pagedir = 'pages' -pagedir = topdir - -# The html prefix which will refer to this directory -# eg. -# httppref = '/newspage/', -# or leave blank for relative links between pages: (Recommended) -# httppref = '' -httppref = '' - -# The name of the 'root' news page in this directory. -# A .html suffix will be added. -rootpage = 'root' - -# Set skipempty to 0 if you wish to see links to empty groups as well. -# Only affects the -a option. -skipempty = 1 - -# pagelinkicon can contain html to put an icon after links to -# further pages. This helps to make important links stand out. -# Set to '' if not wanted, or '...' is quite a good one. -pagelinkicon = '... ' - -# --------------------------------------------------------------------- -# Less important personal preferences: - -# Sublistsize controls the maximum number of items the will appear as -# an indented sub-list before the whole thing is moved onto a different -# page. The smaller this is, the more pages you will have, but the -# shorter each will be. -sublistsize = 4 - -# That should be all. # -####################################################################### - -for dir in os.curdir, os.environ['HOME']: - rcfile = os.path.join(dir, '.newslistrc.py') - if os.path.exists(rcfile): - print rcfile - execfile(rcfile) - break - -from nntplib import NNTP -from stat import * - -rcsrev = '$Revision$' -rcsrev = ' '.join(filter(lambda s: '$' not in s, rcsrev.split())) -desc = {} - -# Make (possibly) relative filenames into absolute ones -treefile = os.path.join(topdir,treefile) -descfile = os.path.join(topdir,descfile) -page = os.path.join(topdir,pagedir) - -# First the bits for creating trees --------------------------- - -# Addtotree creates/augments a tree from a list of group names -def addtotree(tree, groups): - print 'Updating tree...' - for i in groups: - parts = i.split('.') - makeleaf(tree, parts) - -# Makeleaf makes a leaf and the branch leading to it if necessary -def makeleaf(tree,path): - j = path[0] - l = len(path) - - if j not in tree: - tree[j] = {} - if l == 1: - tree[j]['.'] = '.' - if l > 1: - makeleaf(tree[j],path[1:]) - -# Then the bits for outputting trees as pages ---------------- - -# Createpage creates an HTML file named .html containing links -# to those groups beginning with . - -def createpage(root, tree, p): - filename = os.path.join(pagedir, root+'.html') - if root == rootpage: - detail = '' - else: - detail = ' under ' + root - with open(filename, 'w') as f: - # f.write('Content-Type: text/html\n') - f.write('\n\n') - f.write('Newsgroups available%s\n' % detail) - f.write('\n\n') - f.write('

Newsgroups available%s

\n' % detail) - f.write('Back to top level

\n' % - (httppref, rootpage)) - printtree(f, tree, 0, p) - f.write('\n

') - f.write("This page automatically created by 'newslist' v. %s." % - rcsrev) - f.write(time.ctime(time.time()) + '\n') - f.write('\n\n') - -# Printtree prints the groups as a bulleted list. Groups with -# more than subgroups will be put on a separate page. -# Other sets of subgroups are just indented. - -def printtree(f, tree, indent, p): - l = len(tree) - - if l > sublistsize and indent > 0: - # Create a new page and a link to it - f.write('

  • ' % (httppref, p[1:])) - f.write(p[1:] + '.*') - f.write('%s\n' % pagelinkicon) - createpage(p[1:], tree, p) - return - - kl = tree.keys() - - if l > 1: - kl.sort() - if indent > 0: - # Create a sub-list - f.write('
  • %s\n
      ' % p[1:]) - else: - # Create a main list - f.write('
        ') - indent = indent + 1 - - for i in kl: - if i == '.': - # Output a newsgroup - f.write('
      • %s ' % (p[1:], p[1:])) - if p[1:] in desc: - f.write(' %s\n' % desc[p[1:]]) - else: - f.write('\n') - else: - # Output a hierarchy - printtree(f, tree[i], indent, p+'.'+i) - - if l > 1: - f.write('\n
      ') - -# Reading descriptions file --------------------------------------- - -# This returns a dict mapping group name to its description - -def readdesc(descfile): - global desc - desc = {} - - if descfile == '': - return - - try: - with open(descfile, 'r') as d: - print 'Reading descriptions...' - for l in d: - bits = l.split() - try: - grp = bits[0] - dsc = ' '.join(bits[1:]) - if len(dsc) > 1: - desc[grp] = dsc - except IndexError: - pass - except IOError: - print 'Failed to open description file ' + descfile - return - -# Check that ouput directory exists, ------------------------------ -# and offer to create it if not - -def checkopdir(pagedir): - if not os.path.isdir(pagedir): - print 'Directory %s does not exist.' % pagedir - print 'Shall I create it for you? (y/n)' - if sys.stdin.readline()[0] == 'y': - try: - os.mkdir(pagedir, 0777) - except: - print 'Sorry - failed!' - sys.exit(1) - else: - print 'OK. Exiting.' - sys.exit(1) - -# Read and write current local tree ---------------------------------- - -def readlocallist(treefile): - print 'Reading current local group list...' - tree = {} - try: - treetime = time.localtime(os.stat(treefile)[ST_MTIME]) - except: - print '\n*** Failed to open local group cache '+treefile - print 'If this is the first time you have run newslist, then' - print 'use the -a option to create it.' - sys.exit(1) - treedate = '%02d%02d%02d' % (treetime[0] % 100, treetime[1], treetime[2]) - try: - with open(treefile, 'rb') as dump: - tree = marshal.load(dump) - except IOError: - print 'Cannot open local group list ' + treefile - return (tree, treedate) - -def writelocallist(treefile, tree): - try: - with open(treefile, 'wb') as dump: - groups = marshal.dump(tree, dump) - print 'Saved list to %s\n' % treefile - except: - print 'Sorry - failed to write to local group cache', treefile - print 'Does it (or its directory) have the correct permissions?' - sys.exit(1) - -# Return list of all groups on server ----------------------------- - -def getallgroups(server): - print 'Getting list of all groups...' - treedate = '010101' - info = server.list()[1] - groups = [] - print 'Processing...' - if skipempty: - print '\nIgnoring following empty groups:' - for i in info: - grpname = i[0].split()[0] - if skipempty and int(i[1]) < int(i[2]): - print grpname + ' ', - else: - groups.append(grpname) - print '\n' - if skipempty: - print '(End of empty groups)' - return groups - -# Return list of new groups on server ----------------------------- - -def getnewgroups(server, treedate): - print 'Getting list of new groups since start of %s...' % treedate, - info = server.newgroups(treedate, '000001')[1] - print 'got %d.' % len(info) - print 'Processing...', - groups = [] - for i in info: - grpname = i.split()[0] - groups.append(grpname) - print 'Done' - return groups - -# Now the main program -------------------------------------------- - -def main(): - tree = {} - - # Check that the output directory exists - checkopdir(pagedir) - - try: - print 'Connecting to %s...' % newshost - if sys.version[0] == '0': - s = NNTP.init(newshost) - else: - s = NNTP(newshost) - connected = True - except (nntplib.error_temp, nntplib.error_perm), x: - print 'Error connecting to host:', x - print 'I\'ll try to use just the local list.' - connected = False - - # If -a is specified, read the full list of groups from server - if connected and len(sys.argv) > 1 and sys.argv[1] == '-a': - groups = getallgroups(s) - - # Otherwise just read the local file and then add - # groups created since local file last modified. - else: - - (tree, treedate) = readlocallist(treefile) - if connected: - groups = getnewgroups(s, treedate) - - if connected: - addtotree(tree, groups) - writelocallist(treefile,tree) - - # Read group descriptions - readdesc(descfile) - - print 'Creating pages...' - createpage(rootpage, tree, '') - print 'Done' - -if __name__ == "__main__": - main() - -# That's all folks -###################################################################### -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 3 00:45:50 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 02 Dec 2014 23:45:50 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_merge_2=2E7=2E9_release_branch?= Message-ID: <20141202234549.4197.24559@psf.io> https://hg.python.org/cpython/rev/8ec6bf70ed39 changeset: 93708:8ec6bf70ed39 branch: 2.7 parent: 93686:66a261969c57 parent: 93707:54af09408795 user: Benjamin Peterson date: Tue Dec 02 18:45:45 2014 -0500 summary: merge 2.7.9 release branch files: Demo/scripts/newslist.doc | 59 ---- Demo/scripts/newslist.py | 362 -------------------------- 2 files changed, 0 insertions(+), 421 deletions(-) diff --git a/Demo/scripts/newslist.doc b/Demo/scripts/newslist.doc deleted file mode 100644 --- a/Demo/scripts/newslist.doc +++ /dev/null @@ -1,59 +0,0 @@ - NEWSLIST - ======== - A program to assist HTTP browsing of newsgroups - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -WWW browsers such as NCSA Mosaic allow the user to read newsgroup -articles by specifying the group name in a URL eg 'news:comp.answers'. - -To browse through many groups, though, (and there are several thousand -of them) you really need a page or pages containing links to all the -groups. There are some good ones out there, for example, - - http://info.cern.ch/hypertext/DataSources/News/Groups/Overview.html - -is the standard one at CERN, but it only shows the groups available there, -which may be rather different from those available on your machine. - -Newslist is a program which creates a hierarchy of pages for you based -on the groups available from YOUR server. It is written in python - a -splendid interpreted object-oriented language which I suggest you get -right now from the directory /pub/python at ftp.cwi.nl, if you haven't -already got it. - -You should be able to see some sample output by looking at: - http://pelican.cl.cam.ac.uk/newspage/root.html - -Descriptions of newsgroups can be added from a file with one group -per line. eg: - - alt.foo Articles about foo - comp.bar Programming in 'bar' and related languages - -A suitable list detailing most groups can be found at ftp.uu.net in -/uunet-info/newsgroups.gz. - -Make sure you read the information at the beginning of the program source and -configure the variables before running. - -In addition to python, you need: - - An NNTP-based news feed. - A directory in which to put the pages. - -The programming is not very beautiful, but it works! It comes with no -warranty, express or implied, but with the hope that some others may -find it useful. - -Comments, improvements & suggestions welcomed. -Quentin Stafford-Fraser - - ---------------------------------------------------------------------- - Quentin Stafford-Fraser - http://pelican.cl.cam.ac.uk/people/qs101/me.html - - Cambridge University Computer Lab Rank Xerox Cambridge EuroPARC - qs101 at cl.cam.ac.uk fraser at europarc.xerox.com - Tel: +44 223 334411 Tel: +44 223 341521 - Fax: +44 223 334679 Fax: +44 223 341510 - ---------------------------------------------------------------------- diff --git a/Demo/scripts/newslist.py b/Demo/scripts/newslist.py deleted file mode 100755 --- a/Demo/scripts/newslist.py +++ /dev/null @@ -1,362 +0,0 @@ -#! /usr/bin/env python -####################################################################### -# Newslist $Revision$ -# -# Syntax: -# newslist [ -a ] -# -# This is a program to create a directory full of HTML pages -# which between them contain links to all the newsgroups available -# on your server. -# -# The -a option causes a complete list of all groups to be read from -# the server rather than just the ones which have appeared since last -# execution. This recreates the local list from scratch. Use this on -# the first invocation of the program, and from time to time thereafter. -# When new groups are first created they may appear on your server as -# empty groups. By default, empty groups are ignored by the -a option. -# However, these new groups will not be created again, and so will not -# appear in the server's list of 'new groups' at a later date. Hence it -# won't appear until you do a '-a' after some articles have appeared. -# -# I should really keep a list of ignored empty groups and re-check them -# for articles on every run, but I haven't got around to it yet. -# -# This assumes an NNTP news feed. -# -# Feel free to copy, distribute and modify this code for -# non-commercial use. If you make any useful modifications, let me -# know! -# -# (c) Quentin Stafford-Fraser 1994 -# fraser at europarc.xerox.com qs101 at cl.cam.ac.uk -# # -####################################################################### -import sys, nntplib, marshal, time, os - -####################################################################### -# Check these variables before running! # - -# Top directory. -# Filenames which don't start with / are taken as being relative to this. -topdir = os.path.expanduser('~/newspage') - -# The name of your NNTP host -# eg. -# newshost = 'nntp-serv.cl.cam.ac.uk' -# or use following to get the name from the NNTPSERVER environment -# variable: -# newshost = os.environ['NNTPSERVER'] -newshost = 'news.example.com' - -# The filename for a local cache of the newsgroup list -treefile = 'grouptree' - -# The filename for descriptions of newsgroups -# I found a suitable one at ftp.uu.net in /uunet-info/newgroups.gz -# You can set this to '' if you don't wish to use one. -descfile = 'newsgroups' - -# The directory in which HTML pages should be created -# eg. -# pagedir = '/usr/local/lib/html/newspage' -# pagedir = 'pages' -pagedir = topdir - -# The html prefix which will refer to this directory -# eg. -# httppref = '/newspage/', -# or leave blank for relative links between pages: (Recommended) -# httppref = '' -httppref = '' - -# The name of the 'root' news page in this directory. -# A .html suffix will be added. -rootpage = 'root' - -# Set skipempty to 0 if you wish to see links to empty groups as well. -# Only affects the -a option. -skipempty = 1 - -# pagelinkicon can contain html to put an icon after links to -# further pages. This helps to make important links stand out. -# Set to '' if not wanted, or '...' is quite a good one. -pagelinkicon = '... ' - -# --------------------------------------------------------------------- -# Less important personal preferences: - -# Sublistsize controls the maximum number of items the will appear as -# an indented sub-list before the whole thing is moved onto a different -# page. The smaller this is, the more pages you will have, but the -# shorter each will be. -sublistsize = 4 - -# That should be all. # -####################################################################### - -for dir in os.curdir, os.environ['HOME']: - rcfile = os.path.join(dir, '.newslistrc.py') - if os.path.exists(rcfile): - print rcfile - execfile(rcfile) - break - -from nntplib import NNTP -from stat import * - -rcsrev = '$Revision$' -rcsrev = ' '.join(filter(lambda s: '$' not in s, rcsrev.split())) -desc = {} - -# Make (possibly) relative filenames into absolute ones -treefile = os.path.join(topdir,treefile) -descfile = os.path.join(topdir,descfile) -page = os.path.join(topdir,pagedir) - -# First the bits for creating trees --------------------------- - -# Addtotree creates/augments a tree from a list of group names -def addtotree(tree, groups): - print 'Updating tree...' - for i in groups: - parts = i.split('.') - makeleaf(tree, parts) - -# Makeleaf makes a leaf and the branch leading to it if necessary -def makeleaf(tree,path): - j = path[0] - l = len(path) - - if j not in tree: - tree[j] = {} - if l == 1: - tree[j]['.'] = '.' - if l > 1: - makeleaf(tree[j],path[1:]) - -# Then the bits for outputting trees as pages ---------------- - -# Createpage creates an HTML file named .html containing links -# to those groups beginning with . - -def createpage(root, tree, p): - filename = os.path.join(pagedir, root+'.html') - if root == rootpage: - detail = '' - else: - detail = ' under ' + root - with open(filename, 'w') as f: - # f.write('Content-Type: text/html\n') - f.write('\n\n') - f.write('Newsgroups available%s\n' % detail) - f.write('\n\n') - f.write('

      Newsgroups available%s

      \n' % detail) - f.write('Back to top level

      \n' % - (httppref, rootpage)) - printtree(f, tree, 0, p) - f.write('\n

      ') - f.write("This page automatically created by 'newslist' v. %s." % - rcsrev) - f.write(time.ctime(time.time()) + '\n') - f.write('\n\n') - -# Printtree prints the groups as a bulleted list. Groups with -# more than subgroups will be put on a separate page. -# Other sets of subgroups are just indented. - -def printtree(f, tree, indent, p): - l = len(tree) - - if l > sublistsize and indent > 0: - # Create a new page and a link to it - f.write('

    • ' % (httppref, p[1:])) - f.write(p[1:] + '.*') - f.write('%s\n' % pagelinkicon) - createpage(p[1:], tree, p) - return - - kl = tree.keys() - - if l > 1: - kl.sort() - if indent > 0: - # Create a sub-list - f.write('
    • %s\n
        ' % p[1:]) - else: - # Create a main list - f.write('
          ') - indent = indent + 1 - - for i in kl: - if i == '.': - # Output a newsgroup - f.write('
        • %s ' % (p[1:], p[1:])) - if p[1:] in desc: - f.write(' %s\n' % desc[p[1:]]) - else: - f.write('\n') - else: - # Output a hierarchy - printtree(f, tree[i], indent, p+'.'+i) - - if l > 1: - f.write('\n
        ') - -# Reading descriptions file --------------------------------------- - -# This returns a dict mapping group name to its description - -def readdesc(descfile): - global desc - desc = {} - - if descfile == '': - return - - try: - with open(descfile, 'r') as d: - print 'Reading descriptions...' - for l in d: - bits = l.split() - try: - grp = bits[0] - dsc = ' '.join(bits[1:]) - if len(dsc) > 1: - desc[grp] = dsc - except IndexError: - pass - except IOError: - print 'Failed to open description file ' + descfile - return - -# Check that ouput directory exists, ------------------------------ -# and offer to create it if not - -def checkopdir(pagedir): - if not os.path.isdir(pagedir): - print 'Directory %s does not exist.' % pagedir - print 'Shall I create it for you? (y/n)' - if sys.stdin.readline()[0] == 'y': - try: - os.mkdir(pagedir, 0777) - except: - print 'Sorry - failed!' - sys.exit(1) - else: - print 'OK. Exiting.' - sys.exit(1) - -# Read and write current local tree ---------------------------------- - -def readlocallist(treefile): - print 'Reading current local group list...' - tree = {} - try: - treetime = time.localtime(os.stat(treefile)[ST_MTIME]) - except: - print '\n*** Failed to open local group cache '+treefile - print 'If this is the first time you have run newslist, then' - print 'use the -a option to create it.' - sys.exit(1) - treedate = '%02d%02d%02d' % (treetime[0] % 100, treetime[1], treetime[2]) - try: - with open(treefile, 'rb') as dump: - tree = marshal.load(dump) - except IOError: - print 'Cannot open local group list ' + treefile - return (tree, treedate) - -def writelocallist(treefile, tree): - try: - with open(treefile, 'wb') as dump: - groups = marshal.dump(tree, dump) - print 'Saved list to %s\n' % treefile - except: - print 'Sorry - failed to write to local group cache', treefile - print 'Does it (or its directory) have the correct permissions?' - sys.exit(1) - -# Return list of all groups on server ----------------------------- - -def getallgroups(server): - print 'Getting list of all groups...' - treedate = '010101' - info = server.list()[1] - groups = [] - print 'Processing...' - if skipempty: - print '\nIgnoring following empty groups:' - for i in info: - grpname = i[0].split()[0] - if skipempty and int(i[1]) < int(i[2]): - print grpname + ' ', - else: - groups.append(grpname) - print '\n' - if skipempty: - print '(End of empty groups)' - return groups - -# Return list of new groups on server ----------------------------- - -def getnewgroups(server, treedate): - print 'Getting list of new groups since start of %s...' % treedate, - info = server.newgroups(treedate, '000001')[1] - print 'got %d.' % len(info) - print 'Processing...', - groups = [] - for i in info: - grpname = i.split()[0] - groups.append(grpname) - print 'Done' - return groups - -# Now the main program -------------------------------------------- - -def main(): - tree = {} - - # Check that the output directory exists - checkopdir(pagedir) - - try: - print 'Connecting to %s...' % newshost - if sys.version[0] == '0': - s = NNTP.init(newshost) - else: - s = NNTP(newshost) - connected = True - except (nntplib.error_temp, nntplib.error_perm), x: - print 'Error connecting to host:', x - print 'I\'ll try to use just the local list.' - connected = False - - # If -a is specified, read the full list of groups from server - if connected and len(sys.argv) > 1 and sys.argv[1] == '-a': - groups = getallgroups(s) - - # Otherwise just read the local file and then add - # groups created since local file last modified. - else: - - (tree, treedate) = readlocallist(treefile) - if connected: - groups = getnewgroups(s, treedate) - - if connected: - addtotree(tree, groups) - writelocallist(treefile,tree) - - # Read group descriptions - readdesc(descfile) - - print 'Creating pages...' - createpage(rootpage, tree, '') - print 'Done' - -if __name__ == "__main__": - main() - -# That's all folks -###################################################################### -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 3 08:18:40 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 03 Dec 2014 07:18:40 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0MDk5?= =?utf-8?q?=3A_ZipFile=2Eopen=28=29_no_longer_reopen_the_underlying_file?= =?utf-8?q?=2E__Objects?= Message-ID: <20141203071834.105531.59162@psf.io> https://hg.python.org/cpython/rev/c2c4cde55f6f changeset: 93709:c2c4cde55f6f branch: 2.7 user: Serhiy Storchaka date: Wed Dec 03 09:11:12 2014 +0200 summary: Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects returned by ZipFile.open() can now operate independently of the ZipFile even if the ZipFile was created by passing in a file-like object as the first argument to the constructor. files: Doc/library/zipfile.rst | 10 +- Lib/test/test_zipfile.py | 129 +++++++++++++++++++++----- Lib/zipfile.py | 106 +++++++++++++-------- Misc/NEWS | 5 + 4 files changed, 172 insertions(+), 78 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -186,14 +186,8 @@ .. note:: - If the ZipFile was created by passing in a file-like object as the first - argument to the constructor, then the object returned by :meth:`.open` shares the - ZipFile's file pointer. Under these circumstances, the object returned by - :meth:`.open` should not be used after any additional operations are performed - on the ZipFile object. If the ZipFile was created by passing in a string (the - filename) as the first argument to the constructor, then :meth:`.open` will - create a new file object that will be held by the ZipExtFile, allowing it to - operate independently of the ZipFile. + Objects returned by :meth:`.open` can operate independently of the + ZipFile. .. note:: diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -14,7 +14,7 @@ from StringIO import StringIO from tempfile import TemporaryFile -from random import randint, random +from random import randint, random, getrandbits from unittest import skipUnless from test.test_support import TESTFN, TESTFN_UNICODE, TESTFN_ENCODING, \ @@ -35,6 +35,20 @@ ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] +def getrandbytes(size): + return getrandbits(8 * size).to_bytes(size, 'little') + +def getrandbytes(size): + return bytes(bytearray.fromhex('%0*x' % (2 * size, getrandbits(8 * size)))) + +def get_files(test): + yield TESTFN2 + with TemporaryFile() as f: + yield f + test.assertFalse(f.closed) + with io.BytesIO() as f: + yield f + test.assertFalse(f.closed) class TestsWithSourceFile(unittest.TestCase): def setUp(self): @@ -1168,8 +1182,7 @@ # than requested. for test_size in (1, 4095, 4096, 4097, 16384): file_size = test_size + 1 - junk = b''.join(struct.pack('B', randint(0, 255)) - for x in range(file_size)) + junk = getrandbytes(file_size) with zipfile.ZipFile(io.BytesIO(), "w", compression) as zipf: zipf.writestr('foo', junk) with zipf.open('foo', 'r') as fp: @@ -1400,50 +1413,110 @@ @skipUnless(zlib, "requires zlib") class TestsWithMultipleOpens(unittest.TestCase): - def setUp(self): + @classmethod + def setUpClass(cls): + cls.data1 = b'111' + getrandbytes(10000) + cls.data2 = b'222' + getrandbytes(10000) + + def make_test_archive(self, f): # Create the ZIP archive - with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_DEFLATED) as zipfp: - zipfp.writestr('ones', '1'*FIXEDTEST_SIZE) - zipfp.writestr('twos', '2'*FIXEDTEST_SIZE) + with zipfile.ZipFile(f, "w", zipfile.ZIP_DEFLATED) as zipfp: + zipfp.writestr('ones', self.data1) + zipfp.writestr('twos', self.data2) def test_same_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, data2) + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, data2) + self.assertEqual(data1, self.data1) def test_different_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, '1'*FIXEDTEST_SIZE) - self.assertEqual(data2, '2'*FIXEDTEST_SIZE) + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) def test_interleaved(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_read_after_close(self): + for f in get_files(self): + self.make_test_archive(f) + zopen1 = zopen2 = None + try: + with zipfile.ZipFile(f, 'r') as zipf: + zopen1 = zipf.open('ones') + zopen2 = zipf.open('twos') data1 = zopen1.read(500) data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, '1'*FIXEDTEST_SIZE) - self.assertEqual(data2, '2'*FIXEDTEST_SIZE) + data1 += zopen1.read() + data2 += zopen2.read() + finally: + if zopen1: + zopen1.close() + if zopen2: + zopen2.close() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_read_after_write(self): + for f in get_files(self): + with zipfile.ZipFile(f, 'w', zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + zipf.writestr('twos', self.data2) + with zipf.open('ones') as zopen1: + data1 = zopen1.read(500) + self.assertEqual(data1, self.data1[:500]) + with zipfile.ZipFile(f, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_write_after_read(self): + for f in get_files(self): + with zipfile.ZipFile(f, "w", zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + with zipf.open('ones') as zopen1: + zopen1.read(500) + zipf.writestr('twos', self.data2) + with zipfile.ZipFile(f, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) def test_many_opens(self): # Verify that read() and open() promptly close the file descriptor, # and don't rely on the garbage collector to free resources. + self.make_test_archive(TESTFN2) with zipfile.ZipFile(TESTFN2, mode="r") as zipf: for x in range(100): zipf.read('ones') diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -498,6 +498,25 @@ } +class _SharedFile: + def __init__(self, file, pos, close): + self._file = file + self._pos = pos + self._close = close + + def read(self, n=-1): + self._file.seek(self._pos) + data = self._file.read(n) + self._pos = self._file.tell() + return data + + def close(self): + if self._file is not None: + fileobj = self._file + self._file = None + self._close(fileobj) + + class ZipExtFile(io.BufferedIOBase): """File-like object for reading an archive member. Is returned by ZipFile.open(). @@ -743,7 +762,7 @@ self.NameToInfo = {} # Find file info given name self.filelist = [] # List of ZipInfo instances for archive self.compression = compression # Method of compression - self.mode = key = mode.replace('b', '')[0] + self.mode = mode self.pwd = None self._comment = '' @@ -751,28 +770,33 @@ if isinstance(file, basestring): self._filePassed = 0 self.filename = file - modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'} - try: - self.fp = open(file, modeDict[mode]) - except IOError: - if mode == 'a': - mode = key = 'w' - self.fp = open(file, modeDict[mode]) - else: + modeDict = {'r' : 'rb', 'w': 'w+b', 'a' : 'r+b', + 'r+b': 'w+b', 'w+b': 'wb'} + filemode = modeDict[mode] + while True: + try: + self.fp = io.open(file, filemode) + except IOError: + if filemode in modeDict: + filemode = modeDict[filemode] + continue raise + break else: self._filePassed = 1 self.fp = file self.filename = getattr(file, 'name', None) + self._fileRefCnt = 1 try: - if key == 'r': + if mode == 'r': self._RealGetContents() - elif key == 'w': + elif mode == 'w': # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True - elif key == 'a': + self.start_dir = 0 + elif mode == 'a': try: # See if file is a zip file self._RealGetContents() @@ -785,13 +809,13 @@ # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True + self.start_dir = self.fp.tell() else: raise RuntimeError('Mode must be "r", "w" or "a"') except: fp = self.fp self.fp = None - if not self._filePassed: - fp.close() + self._fpclose(fp) raise def __enter__(self): @@ -942,26 +966,17 @@ raise RuntimeError, \ "Attempt to read ZIP archive that was already closed" - # Only open a new file for instances where we were not - # given a file object in the constructor - if self._filePassed: - zef_file = self.fp - should_close = False + # Make sure we have an info object + if isinstance(name, ZipInfo): + # 'name' is already an info object + zinfo = name else: - zef_file = open(self.filename, 'rb') - should_close = True + # Get info object for name + zinfo = self.getinfo(name) + self._fileRefCnt += 1 + zef_file = _SharedFile(self.fp, zinfo.header_offset, self._fpclose) try: - # Make sure we have an info object - if isinstance(name, ZipInfo): - # 'name' is already an info object - zinfo = name - else: - # Get info object for name - zinfo = self.getinfo(name) - - zef_file.seek(zinfo.header_offset, 0) - # Skip the file header: fheader = zef_file.read(sizeFileHeader) if len(fheader) != sizeFileHeader: @@ -1006,11 +1021,9 @@ if ord(h[11]) != check_byte: raise RuntimeError("Bad password for file", name) - return ZipExtFile(zef_file, mode, zinfo, zd, - close_fileobj=should_close) + return ZipExtFile(zef_file, mode, zinfo, zd, True) except: - if should_close: - zef_file.close() + zef_file.close() raise def extract(self, member, path=None, pwd=None): @@ -1141,6 +1154,7 @@ zinfo.file_size = st.st_size zinfo.flag_bits = 0x00 + self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header bytes self._writecheck(zinfo) @@ -1154,6 +1168,7 @@ self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo self.fp.write(zinfo.FileHeader(False)) + self.start_dir = self.fp.tell() return with open(filename, "rb") as fp: @@ -1196,10 +1211,10 @@ raise RuntimeError('Compressed size larger than uncompressed size') # Seek backwards and write file header (which will now include # correct CRC and file sizes) - position = self.fp.tell() # Preserve current position in file + self.start_dir = self.fp.tell() # Preserve current position in file self.fp.seek(zinfo.header_offset, 0) self.fp.write(zinfo.FileHeader(zip64)) - self.fp.seek(position, 0) + self.fp.seek(self.start_dir, 0) self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1228,6 +1243,7 @@ zinfo.compress_type = compress_type zinfo.file_size = len(bytes) # Uncompressed size + self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header bytes self._writecheck(zinfo) self._didModify = True @@ -1251,6 +1267,7 @@ self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size, zinfo.file_size)) self.fp.flush() + self.start_dir = self.fp.tell() self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1266,7 +1283,7 @@ try: if self.mode in ("w", "a") and self._didModify: # write ending records - pos1 = self.fp.tell() + self.fp.seek(self.start_dir, 0) for zinfo in self.filelist: # write central directory dt = zinfo.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -1329,8 +1346,8 @@ pos2 = self.fp.tell() # Write end-of-zip-archive record centDirCount = len(self.filelist) - centDirSize = pos2 - pos1 - centDirOffset = pos1 + centDirSize = pos2 - self.start_dir + centDirOffset = self.start_dir requires_zip64 = None if centDirCount > ZIP_FILECOUNT_LIMIT: requires_zip64 = "Files count" @@ -1366,8 +1383,13 @@ finally: fp = self.fp self.fp = None - if not self._filePassed: - fp.close() + self._fpclose(fp) + + def _fpclose(self, fp): + assert self._fileRefCnt > 0 + self._fileRefCnt -= 1 + if not self._fileRefCnt and not self._filePassed: + fp.close() class PyZipFile(ZipFile): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,11 @@ Library ------- +- Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects + returned by ZipFile.open() can now operate independently of the ZipFile even + if the ZipFile was created by passing in a file-like object as the first + argument to the constructor. + - Issue #21032. Fixed socket leak if HTTPConnection.getresponse() fails. Original patch by Martin Panter. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 3 08:18:40 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 03 Dec 2014 07:18:40 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2314099=3A_ZipFile=2Eopen=28=29_no_longer_reopen_?= =?utf-8?q?the_underlying_file=2E__Objects?= Message-ID: <20141203071835.96692.76725@psf.io> https://hg.python.org/cpython/rev/334c01aa7f93 changeset: 93711:334c01aa7f93 parent: 93706:fd80195b920f parent: 93710:e5bb3044402b user: Serhiy Storchaka date: Wed Dec 03 09:14:36 2014 +0200 summary: Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects returned by ZipFile.open() can now operate independently of the ZipFile even if the ZipFile was created by passing in a file-like object as the first argument to the constructor. files: Doc/library/zipfile.rst | 10 +- Lib/test/test_zipfile.py | 121 +++++++++++++++++++++----- Lib/zipfile.py | 103 ++++++++++++++-------- Misc/NEWS | 5 + 4 files changed, 166 insertions(+), 73 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -219,14 +219,8 @@ .. note:: - If the ZipFile was created by passing in a file-like object as the first - argument to the constructor, then the object returned by :meth:`.open` shares the - ZipFile's file pointer. Under these circumstances, the object returned by - :meth:`.open` should not be used after any additional operations are performed - on the ZipFile object. If the ZipFile was created by passing in a string (the - filename) as the first argument to the constructor, then :meth:`.open` will - create a new file object that will be held by the ZipExtFile, allowing it to - operate independently of the ZipFile. + Objects returned by :meth:`.open` can operate independently of the + ZipFile. .. note:: diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -1,3 +1,4 @@ +import contextlib import io import os import sys @@ -25,6 +26,9 @@ ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] +def getrandbytes(size): + return getrandbits(8 * size).to_bytes(size, 'little') + def get_files(test): yield TESTFN2 with TemporaryFile() as f: @@ -289,7 +293,7 @@ # than requested. for test_size in (1, 4095, 4096, 4097, 16384): file_size = test_size + 1 - junk = getrandbits(8 * file_size).to_bytes(file_size, 'little') + junk = getrandbytes(file_size) with zipfile.ZipFile(io.BytesIO(), "w", self.compression) as zipf: zipf.writestr('foo', junk) with zipf.open('foo', 'r') as fp: @@ -1666,46 +1670,111 @@ @requires_zlib class TestsWithMultipleOpens(unittest.TestCase): - def setUp(self): + @classmethod + def setUpClass(cls): + cls.data1 = b'111' + getrandbytes(10000) + cls.data2 = b'222' + getrandbytes(10000) + + def make_test_archive(self, f): # Create the ZIP archive - with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_DEFLATED) as zipfp: - zipfp.writestr('ones', '1'*FIXEDTEST_SIZE) - zipfp.writestr('twos', '2'*FIXEDTEST_SIZE) + with zipfile.ZipFile(f, "w", zipfile.ZIP_DEFLATED) as zipfp: + zipfp.writestr('ones', self.data1) + zipfp.writestr('twos', self.data2) def test_same_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, data2) + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, data2) + self.assertEqual(data1, self.data1) def test_different_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, b'1'*FIXEDTEST_SIZE) - self.assertEqual(data2, b'2'*FIXEDTEST_SIZE) + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) def test_interleaved(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_read_after_close(self): + for f in get_files(self): + self.make_test_archive(f) + with contextlib.ExitStack() as stack: + with zipfile.ZipFile(f, 'r') as zipf: + zopen1 = stack.enter_context(zipf.open('ones')) + zopen2 = stack.enter_context(zipf.open('twos')) data1 = zopen1.read(500) data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, b'1'*FIXEDTEST_SIZE) - self.assertEqual(data2, b'2'*FIXEDTEST_SIZE) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_read_after_write(self): + for f in get_files(self): + with zipfile.ZipFile(f, 'w', zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + zipf.writestr('twos', self.data2) + with zipf.open('ones') as zopen1: + data1 = zopen1.read(500) + self.assertEqual(data1, self.data1[:500]) + with zipfile.ZipFile(f, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_write_after_read(self): + for f in get_files(self): + with zipfile.ZipFile(f, "w", zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + with zipf.open('ones') as zopen1: + zopen1.read(500) + zipf.writestr('twos', self.data2) + with zipfile.ZipFile(f, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_many_opens(self): + # Verify that read() and open() promptly close the file descriptor, + # and don't rely on the garbage collector to free resources. + self.make_test_archive(TESTFN2) + with zipfile.ZipFile(TESTFN2, mode="r") as zipf: + for x in range(100): + zipf.read('ones') + with zipf.open('ones') as zopen1: + pass + with open(os.devnull) as f: + self.assertLess(f.fileno(), 100) def tearDown(self): unlink(TESTFN2) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -646,6 +646,25 @@ raise NotImplementedError("compression type %d" % (compress_type,)) +class _SharedFile: + def __init__(self, file, pos, close): + self._file = file + self._pos = pos + self._close = close + + def read(self, n=-1): + self._file.seek(self._pos) + data = self._file.read(n) + self._pos = self._file.tell() + return data + + def close(self): + if self._file is not None: + fileobj = self._file + self._file = None + self._close(fileobj) + + class ZipExtFile(io.BufferedIOBase): """File-like object for reading an archive member. Is returned by ZipFile.open(). @@ -945,7 +964,7 @@ self.NameToInfo = {} # Find file info given name self.filelist = [] # List of ZipInfo instances for archive self.compression = compression # Method of compression - self.mode = key = mode.replace('b', '')[0] + self.mode = mode self.pwd = None self._comment = b'' @@ -954,28 +973,33 @@ # No, it's a filename self._filePassed = 0 self.filename = file - modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'} - try: - self.fp = io.open(file, modeDict[mode]) - except OSError: - if mode == 'a': - mode = key = 'w' - self.fp = io.open(file, modeDict[mode]) - else: + modeDict = {'r' : 'rb', 'w': 'w+b', 'a' : 'r+b', + 'r+b': 'w+b', 'w+b': 'wb'} + filemode = modeDict[mode] + while True: + try: + self.fp = io.open(file, filemode) + except OSError: + if filemode in modeDict: + filemode = modeDict[filemode] + continue raise + break else: self._filePassed = 1 self.fp = file self.filename = getattr(file, 'name', None) + self._fileRefCnt = 1 try: - if key == 'r': + if mode == 'r': self._RealGetContents() - elif key == 'w': + elif mode == 'w': # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True - elif key == 'a': + self.start_dir = 0 + elif mode == 'a': try: # See if file is a zip file self._RealGetContents() @@ -988,13 +1012,13 @@ # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True + self.start_dir = self.fp.tell() else: raise RuntimeError('Mode must be "r", "w" or "a"') except: fp = self.fp self.fp = None - if not self._filePassed: - fp.close() + self._fpclose(fp) raise def __enter__(self): @@ -1181,23 +1205,17 @@ raise RuntimeError( "Attempt to read ZIP archive that was already closed") - # Only open a new file for instances where we were not - # given a file object in the constructor - if self._filePassed: - zef_file = self.fp + # Make sure we have an info object + if isinstance(name, ZipInfo): + # 'name' is already an info object + zinfo = name else: - zef_file = io.open(self.filename, 'rb') + # Get info object for name + zinfo = self.getinfo(name) + self._fileRefCnt += 1 + zef_file = _SharedFile(self.fp, zinfo.header_offset, self._fpclose) try: - # Make sure we have an info object - if isinstance(name, ZipInfo): - # 'name' is already an info object - zinfo = name - else: - # Get info object for name - zinfo = self.getinfo(name) - zef_file.seek(zinfo.header_offset, 0) - # Skip the file header: fheader = zef_file.read(sizeFileHeader) if len(fheader) != sizeFileHeader: @@ -1256,11 +1274,9 @@ if h[11] != check_byte: raise RuntimeError("Bad password for file", name) - return ZipExtFile(zef_file, mode, zinfo, zd, - close_fileobj=not self._filePassed) + return ZipExtFile(zef_file, mode, zinfo, zd, True) except: - if not self._filePassed: - zef_file.close() + zef_file.close() raise def extract(self, member, path=None, pwd=None): @@ -1394,6 +1410,7 @@ zinfo.file_size = st.st_size zinfo.flag_bits = 0x00 + self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header bytes if zinfo.compress_type == ZIP_LZMA: # Compressed data includes an end-of-stream (EOS) marker @@ -1410,6 +1427,7 @@ self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo self.fp.write(zinfo.FileHeader(False)) + self.start_dir = self.fp.tell() return cmpr = _get_compressor(zinfo.compress_type) @@ -1448,10 +1466,10 @@ raise RuntimeError('Compressed size larger than uncompressed size') # Seek backwards and write file header (which will now include # correct CRC and file sizes) - position = self.fp.tell() # Preserve current position in file + self.start_dir = self.fp.tell() # Preserve current position in file self.fp.seek(zinfo.header_offset, 0) self.fp.write(zinfo.FileHeader(zip64)) - self.fp.seek(position, 0) + self.fp.seek(self.start_dir, 0) self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1480,6 +1498,7 @@ "Attempt to write to ZIP archive that was already closed") zinfo.file_size = len(data) # Uncompressed size + self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header data if compress_type is not None: zinfo.compress_type = compress_type @@ -1508,6 +1527,7 @@ self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size, zinfo.file_size)) self.fp.flush() + self.start_dir = self.fp.tell() self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1523,7 +1543,7 @@ try: if self.mode in ("w", "a") and self._didModify: # write ending records - pos1 = self.fp.tell() + self.fp.seek(self.start_dir, 0) for zinfo in self.filelist: # write central directory dt = zinfo.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -1589,8 +1609,8 @@ pos2 = self.fp.tell() # Write end-of-zip-archive record centDirCount = len(self.filelist) - centDirSize = pos2 - pos1 - centDirOffset = pos1 + centDirSize = pos2 - self.start_dir + centDirOffset = self.start_dir requires_zip64 = None if centDirCount > ZIP_FILECOUNT_LIMIT: requires_zip64 = "Files count" @@ -1626,8 +1646,13 @@ finally: fp = self.fp self.fp = None - if not self._filePassed: - fp.close() + self._fpclose(fp) + + def _fpclose(self, fp): + assert self._fileRefCnt > 0 + self._fileRefCnt -= 1 + if not self._fileRefCnt and not self._filePassed: + fp.close() class PyZipFile(ZipFile): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -194,6 +194,11 @@ Library ------- +- Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects + returned by ZipFile.open() can now operate independently of the ZipFile even + if the ZipFile was created by passing in a file-like object as the first + argument to the constructor. + - Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is asked to compile a source file containing multiple dots in the source file name. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 3 08:18:40 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 03 Dec 2014 07:18:40 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE0MDk5?= =?utf-8?q?=3A_ZipFile=2Eopen=28=29_no_longer_reopen_the_underlying_file?= =?utf-8?q?=2E__Objects?= Message-ID: <20141203071835.96692.46884@psf.io> https://hg.python.org/cpython/rev/e5bb3044402b changeset: 93710:e5bb3044402b branch: 3.4 parent: 93704:4b6b03c1f4ff user: Serhiy Storchaka date: Wed Dec 03 09:11:57 2014 +0200 summary: Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects returned by ZipFile.open() can now operate independently of the ZipFile even if the ZipFile was created by passing in a file-like object as the first argument to the constructor. files: Doc/library/zipfile.rst | 10 +- Lib/test/test_zipfile.py | 121 +++++++++++++++++++++----- Lib/zipfile.py | 103 ++++++++++++++-------- Misc/NEWS | 5 + 4 files changed, 166 insertions(+), 73 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -219,14 +219,8 @@ .. note:: - If the ZipFile was created by passing in a file-like object as the first - argument to the constructor, then the object returned by :meth:`.open` shares the - ZipFile's file pointer. Under these circumstances, the object returned by - :meth:`.open` should not be used after any additional operations are performed - on the ZipFile object. If the ZipFile was created by passing in a string (the - filename) as the first argument to the constructor, then :meth:`.open` will - create a new file object that will be held by the ZipExtFile, allowing it to - operate independently of the ZipFile. + Objects returned by :meth:`.open` can operate independently of the + ZipFile. .. note:: diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -1,3 +1,4 @@ +import contextlib import io import os import sys @@ -25,6 +26,9 @@ ('ziptest2dir/ziptest3dir/_ziptest3', 'azsxdcfvgb'), ('ziptest2dir/ziptest3dir/ziptest4dir/_ziptest3', '6y7u8i9o0p')] +def getrandbytes(size): + return getrandbits(8 * size).to_bytes(size, 'little') + def get_files(test): yield TESTFN2 with TemporaryFile() as f: @@ -289,7 +293,7 @@ # than requested. for test_size in (1, 4095, 4096, 4097, 16384): file_size = test_size + 1 - junk = getrandbits(8 * file_size).to_bytes(file_size, 'little') + junk = getrandbytes(file_size) with zipfile.ZipFile(io.BytesIO(), "w", self.compression) as zipf: zipf.writestr('foo', junk) with zipf.open('foo', 'r') as fp: @@ -1635,46 +1639,111 @@ @requires_zlib class TestsWithMultipleOpens(unittest.TestCase): - def setUp(self): + @classmethod + def setUpClass(cls): + cls.data1 = b'111' + getrandbytes(10000) + cls.data2 = b'222' + getrandbytes(10000) + + def make_test_archive(self, f): # Create the ZIP archive - with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_DEFLATED) as zipfp: - zipfp.writestr('ones', '1'*FIXEDTEST_SIZE) - zipfp.writestr('twos', '2'*FIXEDTEST_SIZE) + with zipfile.ZipFile(f, "w", zipfile.ZIP_DEFLATED) as zipfp: + zipfp.writestr('ones', self.data1) + zipfp.writestr('twos', self.data2) def test_same_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, data2) + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('ones') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, data2) + self.assertEqual(data1, self.data1) def test_different_file(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: - data1 = zopen1.read(500) - data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, b'1'*FIXEDTEST_SIZE) - self.assertEqual(data2, b'2'*FIXEDTEST_SIZE) + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) def test_interleaved(self): # Verify that (when the ZipFile is in control of creating file objects) # multiple open() calls can be made without interfering with each other. - with zipfile.ZipFile(TESTFN2, mode="r") as zipf: - with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + for f in get_files(self): + self.make_test_archive(f) + with zipfile.ZipFile(f, mode="r") as zipf: + with zipf.open('ones') as zopen1, zipf.open('twos') as zopen2: + data1 = zopen1.read(500) + data2 = zopen2.read(500) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_read_after_close(self): + for f in get_files(self): + self.make_test_archive(f) + with contextlib.ExitStack() as stack: + with zipfile.ZipFile(f, 'r') as zipf: + zopen1 = stack.enter_context(zipf.open('ones')) + zopen2 = stack.enter_context(zipf.open('twos')) data1 = zopen1.read(500) data2 = zopen2.read(500) - data1 += zopen1.read(500) - data2 += zopen2.read(500) - self.assertEqual(data1, b'1'*FIXEDTEST_SIZE) - self.assertEqual(data2, b'2'*FIXEDTEST_SIZE) + data1 += zopen1.read() + data2 += zopen2.read() + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_read_after_write(self): + for f in get_files(self): + with zipfile.ZipFile(f, 'w', zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + zipf.writestr('twos', self.data2) + with zipf.open('ones') as zopen1: + data1 = zopen1.read(500) + self.assertEqual(data1, self.data1[:500]) + with zipfile.ZipFile(f, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_write_after_read(self): + for f in get_files(self): + with zipfile.ZipFile(f, "w", zipfile.ZIP_DEFLATED) as zipf: + zipf.writestr('ones', self.data1) + with zipf.open('ones') as zopen1: + zopen1.read(500) + zipf.writestr('twos', self.data2) + with zipfile.ZipFile(f, 'r') as zipf: + data1 = zipf.read('ones') + data2 = zipf.read('twos') + self.assertEqual(data1, self.data1) + self.assertEqual(data2, self.data2) + + def test_many_opens(self): + # Verify that read() and open() promptly close the file descriptor, + # and don't rely on the garbage collector to free resources. + self.make_test_archive(TESTFN2) + with zipfile.ZipFile(TESTFN2, mode="r") as zipf: + for x in range(100): + zipf.read('ones') + with zipf.open('ones') as zopen1: + pass + with open(os.devnull) as f: + self.assertLess(f.fileno(), 100) def tearDown(self): unlink(TESTFN2) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -624,6 +624,25 @@ raise NotImplementedError("compression type %d" % (compress_type,)) +class _SharedFile: + def __init__(self, file, pos, close): + self._file = file + self._pos = pos + self._close = close + + def read(self, n=-1): + self._file.seek(self._pos) + data = self._file.read(n) + self._pos = self._file.tell() + return data + + def close(self): + if self._file is not None: + fileobj = self._file + self._file = None + self._close(fileobj) + + class ZipExtFile(io.BufferedIOBase): """File-like object for reading an archive member. Is returned by ZipFile.open(). @@ -909,7 +928,7 @@ self.NameToInfo = {} # Find file info given name self.filelist = [] # List of ZipInfo instances for archive self.compression = compression # Method of compression - self.mode = key = mode.replace('b', '')[0] + self.mode = mode self.pwd = None self._comment = b'' @@ -918,28 +937,33 @@ # No, it's a filename self._filePassed = 0 self.filename = file - modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'} - try: - self.fp = io.open(file, modeDict[mode]) - except OSError: - if mode == 'a': - mode = key = 'w' - self.fp = io.open(file, modeDict[mode]) - else: + modeDict = {'r' : 'rb', 'w': 'w+b', 'a' : 'r+b', + 'r+b': 'w+b', 'w+b': 'wb'} + filemode = modeDict[mode] + while True: + try: + self.fp = io.open(file, filemode) + except OSError: + if filemode in modeDict: + filemode = modeDict[filemode] + continue raise + break else: self._filePassed = 1 self.fp = file self.filename = getattr(file, 'name', None) + self._fileRefCnt = 1 try: - if key == 'r': + if mode == 'r': self._RealGetContents() - elif key == 'w': + elif mode == 'w': # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True - elif key == 'a': + self.start_dir = 0 + elif mode == 'a': try: # See if file is a zip file self._RealGetContents() @@ -952,13 +976,13 @@ # set the modified flag so central directory gets written # even if no files are added to the archive self._didModify = True + self.start_dir = self.fp.tell() else: raise RuntimeError('Mode must be "r", "w" or "a"') except: fp = self.fp self.fp = None - if not self._filePassed: - fp.close() + self._fpclose(fp) raise def __enter__(self): @@ -1131,23 +1155,17 @@ raise RuntimeError( "Attempt to read ZIP archive that was already closed") - # Only open a new file for instances where we were not - # given a file object in the constructor - if self._filePassed: - zef_file = self.fp + # Make sure we have an info object + if isinstance(name, ZipInfo): + # 'name' is already an info object + zinfo = name else: - zef_file = io.open(self.filename, 'rb') + # Get info object for name + zinfo = self.getinfo(name) + self._fileRefCnt += 1 + zef_file = _SharedFile(self.fp, zinfo.header_offset, self._fpclose) try: - # Make sure we have an info object - if isinstance(name, ZipInfo): - # 'name' is already an info object - zinfo = name - else: - # Get info object for name - zinfo = self.getinfo(name) - zef_file.seek(zinfo.header_offset, 0) - # Skip the file header: fheader = zef_file.read(sizeFileHeader) if len(fheader) != sizeFileHeader: @@ -1206,11 +1224,9 @@ if h[11] != check_byte: raise RuntimeError("Bad password for file", name) - return ZipExtFile(zef_file, mode, zinfo, zd, - close_fileobj=not self._filePassed) + return ZipExtFile(zef_file, mode, zinfo, zd, True) except: - if not self._filePassed: - zef_file.close() + zef_file.close() raise def extract(self, member, path=None, pwd=None): @@ -1344,6 +1360,7 @@ zinfo.file_size = st.st_size zinfo.flag_bits = 0x00 + self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header bytes if zinfo.compress_type == ZIP_LZMA: # Compressed data includes an end-of-stream (EOS) marker @@ -1360,6 +1377,7 @@ self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo self.fp.write(zinfo.FileHeader(False)) + self.start_dir = self.fp.tell() return cmpr = _get_compressor(zinfo.compress_type) @@ -1398,10 +1416,10 @@ raise RuntimeError('Compressed size larger than uncompressed size') # Seek backwards and write file header (which will now include # correct CRC and file sizes) - position = self.fp.tell() # Preserve current position in file + self.start_dir = self.fp.tell() # Preserve current position in file self.fp.seek(zinfo.header_offset, 0) self.fp.write(zinfo.FileHeader(zip64)) - self.fp.seek(position, 0) + self.fp.seek(self.start_dir, 0) self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1430,6 +1448,7 @@ "Attempt to write to ZIP archive that was already closed") zinfo.file_size = len(data) # Uncompressed size + self.fp.seek(self.start_dir, 0) zinfo.header_offset = self.fp.tell() # Start of header data if compress_type is not None: zinfo.compress_type = compress_type @@ -1458,6 +1477,7 @@ self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size, zinfo.file_size)) self.fp.flush() + self.start_dir = self.fp.tell() self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1473,7 +1493,7 @@ try: if self.mode in ("w", "a") and self._didModify: # write ending records - pos1 = self.fp.tell() + self.fp.seek(self.start_dir, 0) for zinfo in self.filelist: # write central directory dt = zinfo.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -1539,8 +1559,8 @@ pos2 = self.fp.tell() # Write end-of-zip-archive record centDirCount = len(self.filelist) - centDirSize = pos2 - pos1 - centDirOffset = pos1 + centDirSize = pos2 - self.start_dir + centDirOffset = self.start_dir requires_zip64 = None if centDirCount > ZIP_FILECOUNT_LIMIT: requires_zip64 = "Files count" @@ -1576,8 +1596,13 @@ finally: fp = self.fp self.fp = None - if not self._filePassed: - fp.close() + self._fpclose(fp) + + def _fpclose(self, fp): + assert self._fileRefCnt > 0 + self._fileRefCnt -= 1 + if not self._fileRefCnt and not self._filePassed: + fp.close() class PyZipFile(ZipFile): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -39,6 +39,11 @@ Library ------- +- Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects + returned by ZipFile.open() can now operate independently of the ZipFile even + if the ZipFile was created by passing in a file-like object as the first + argument to the constructor. + - Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is asked to compile a source file containing multiple dots in the source file name. -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Wed Dec 3 10:32:35 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 03 Dec 2014 10:32:35 +0100 Subject: [Python-checkins] Daily reference leaks (fd80195b920f): sum=3 Message-ID: results for fd80195b920f on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogyBtMc4', '-x'] From python-checkins at python.org Wed Dec 3 20:04:05 2014 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 03 Dec 2014 19:04:05 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogRml4ICMyMjk4Nzog?= =?utf-8?q?update_the_compatibility_matrix_for_a_SSLv23_client=2E?= Message-ID: <20141203190355.4211.91714@psf.io> https://hg.python.org/cpython/rev/7af5d5493497 changeset: 93712:7af5d5493497 branch: 2.7 parent: 93709:c2c4cde55f6f user: Antoine Pitrou date: Wed Dec 03 20:00:56 2014 +0100 summary: Fix #22987: update the compatibility matrix for a SSLv23 client. files: Doc/library/ssl.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -192,7 +192,7 @@ ------------------------ --------- --------- ---------- --------- ----------- ----------- *SSLv2* yes no yes no no no *SSLv3* no yes yes no no no - *SSLv23* yes no yes no no no + *SSLv23* no yes yes yes yes yes *TLSv1* no no yes yes no no *TLSv1.1* no no yes no yes no *TLSv1.2* no no yes no no yes @@ -201,9 +201,8 @@ .. note:: Which connections succeed will vary depending on the version of - OpenSSL. For example, beginning with OpenSSL 1.0.0, an SSLv23 client - will not actually attempt SSLv2 connections unless you explicitly - enable SSLv2 ciphers (which is not recommended, as SSLv2 is broken). + OpenSSL. For example, before OpenSSL 1.0.0, an SSLv23 client + would always attempt SSLv2 connections. The *ciphers* parameter sets the available ciphers for this SSL object. It should be a string in the `OpenSSL cipher list format -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 3 20:04:05 2014 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 03 Dec 2014 19:04:05 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Fix_=2322987=3A_update_the_compatibility_matrix_for_a_SS?= =?utf-8?q?Lv23_client=2E?= Message-ID: <20141203190355.96698.95199@psf.io> https://hg.python.org/cpython/rev/7509a0607c40 changeset: 93714:7509a0607c40 parent: 93711:334c01aa7f93 parent: 93713:9f03572690d2 user: Antoine Pitrou date: Wed Dec 03 20:03:11 2014 +0100 summary: Fix #22987: update the compatibility matrix for a SSLv23 client. files: Doc/library/ssl.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -192,7 +192,7 @@ ------------------------ --------- --------- ---------- --------- ----------- ----------- *SSLv2* yes no yes no no no *SSLv3* no yes yes no no no - *SSLv23* yes no yes no no no + *SSLv23* no yes yes yes yes yes *TLSv1* no no yes yes no no *TLSv1.1* no no yes no yes no *TLSv1.2* no no yes no no yes @@ -201,9 +201,8 @@ .. note:: Which connections succeed will vary depending on the version of - OpenSSL. For example, beginning with OpenSSL 1.0.0, an SSLv23 client - will not actually attempt SSLv2 connections unless you explicitly - enable SSLv2 ciphers (which is not recommended, as SSLv2 is broken). + OpenSSL. For example, before OpenSSL 1.0.0, an SSLv23 client + would always attempt SSLv2 connections. The *ciphers* parameter sets the available ciphers for this SSL object. It should be a string in the `OpenSSL cipher list format -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 3 20:04:07 2014 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 03 Dec 2014 19:04:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogRml4ICMyMjk4Nzog?= =?utf-8?q?update_the_compatibility_matrix_for_a_SSLv23_client=2E?= Message-ID: <20141203190355.22680.33248@psf.io> https://hg.python.org/cpython/rev/9f03572690d2 changeset: 93713:9f03572690d2 branch: 3.4 parent: 93710:e5bb3044402b user: Antoine Pitrou date: Wed Dec 03 20:00:56 2014 +0100 summary: Fix #22987: update the compatibility matrix for a SSLv23 client. files: Doc/library/ssl.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -192,7 +192,7 @@ ------------------------ --------- --------- ---------- --------- ----------- ----------- *SSLv2* yes no yes no no no *SSLv3* no yes yes no no no - *SSLv23* yes no yes no no no + *SSLv23* no yes yes yes yes yes *TLSv1* no no yes yes no no *TLSv1.1* no no yes no yes no *TLSv1.2* no no yes no no yes @@ -201,9 +201,8 @@ .. note:: Which connections succeed will vary depending on the version of - OpenSSL. For example, beginning with OpenSSL 1.0.0, an SSLv23 client - will not actually attempt SSLv2 connections unless you explicitly - enable SSLv2 ciphers (which is not recommended, as SSLv2 is broken). + OpenSSL. For example, before OpenSSL 1.0.0, an SSLv23 client + would always attempt SSLv2 connections. The *ciphers* parameter sets the available ciphers for this SSL object. It should be a string in the `OpenSSL cipher list format -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 06:56:11 2014 From: python-checkins at python.org (terry.reedy) Date: Thu, 04 Dec 2014 05:56:11 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE2ODkz?= =?utf-8?q?=3A_For_Idle_doc=2C_move_index_entries=2C_copy_no-subprocess_se?= =?utf-8?q?ction?= Message-ID: <20141204055607.9545.75268@psf.io> https://hg.python.org/cpython/rev/6db65ff985b6 changeset: 93715:6db65ff985b6 branch: 3.4 parent: 93713:9f03572690d2 user: Terry Jan Reedy date: Thu Dec 04 00:54:59 2014 -0500 summary: Issue #16893: For Idle doc, move index entries, copy no-subprocess section from idlelib/help.txt, add 'C' comment, and remove obsolete ^Z comment Original patch by Zach Ware. files: Doc/library/idle.rst | 58 ++++++++++++++++++++----------- 1 files changed, 38 insertions(+), 20 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -1,15 +1,15 @@ .. _idle: +.. index:: + single: IDLE + single: Python Editor + single: Integrated Development Environment + IDLE ==== .. moduleauthor:: Guido van Rossum -.. index:: - single: IDLE - single: Python Editor - single: Integrated Development Environment - IDLE is the Python IDE built with the :mod:`tkinter` GUI toolkit. IDLE has the following features: @@ -51,16 +51,16 @@ Recent Files Open a list of recent files +.. index:: + single: Class browser + single: Path browser + Class browser Show classes and methods in current file Path browser Show sys.path directories, modules, classes and methods -.. index:: - single: Class browser - single: Path browser - Save Save current window to the associated file (unsaved windows have a \* before and after the window title) @@ -137,7 +137,7 @@ Format menu (Editor window only) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Indent region Shift selected lines right by the indent width (default 4 spaces) @@ -263,6 +263,14 @@ Additional help sources may be added here with the Configure IDLE dialog under the General tab. +.. index:: + single: Cut + single: Copy + single: Paste + single: Set Breakpoint + single: Clear Breakpoint + single: breakpoints + Editor Window context menu ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -283,15 +291,6 @@ Clear Breakpoint Clears the breakpoint on that line. -.. index:: - single: Cut - single: Copy - single: Paste - single: Set Breakpoint - single: Clear Breakpoint - single: breakpoints - - Shell Window context menu ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -313,6 +312,9 @@ Editing and navigation ---------------------- +In this section, 'C' refers to the Control key on Windows and Unix and +the Command key on Mac OSX. + * :kbd:`Backspace` deletes to the left; :kbd:`Del` deletes to the right * :kbd:`C-Backspace` delete word left; :kbd:`C-Del` delete word to the right @@ -414,7 +416,6 @@ * :kbd:`C-c` interrupts executing command * :kbd:`C-d` sends end-of-file; closes window if typed at a ``>>>`` prompt - (this is :kbd:`C-z` on Windows). * :kbd:`Alt-/` (Expand word) is also useful to reduce typing @@ -507,6 +508,22 @@ name is '-', no script is executed but an interactive Python session is started; the arguments are still available in ``sys.argv``. +Running without a subprocess +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If IDLE is started with the -n command line switch it will run in a +single process and will not create the subprocess which runs the RPC +Python execution server. This can be useful if Python cannot create +the subprocess or the RPC socket interface on your platform. However, +in this mode user code is not isolated from IDLE itself. Also, the +environment is not restarted when Run/Run Module (F5) is selected. If +your code has been modified, you must reload() the affected modules and +re-import any specific items (e.g. from foo import baz) if the changes +are to take effect. For these reasons, it is preferable to run IDLE +with the default subprocess if at all possible. + +.. deprecated:: 3.4 + Additional help sources ----------------------- @@ -527,6 +544,7 @@ addition a user can create a custom key set in the Configure IDLE dialog under the keys tab. + Extensions ---------- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 06:56:11 2014 From: python-checkins at python.org (terry.reedy) Date: Thu, 04 Dec 2014 05:56:11 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <20141204055607.9555.65279@psf.io> https://hg.python.org/cpython/rev/dc146e018fe3 changeset: 93716:dc146e018fe3 parent: 93714:7509a0607c40 parent: 93715:6db65ff985b6 user: Terry Jan Reedy date: Thu Dec 04 00:55:16 2014 -0500 summary: Merge with 3.4 files: Doc/library/idle.rst | 58 ++++++++++++++++++++----------- 1 files changed, 38 insertions(+), 20 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -1,15 +1,15 @@ .. _idle: +.. index:: + single: IDLE + single: Python Editor + single: Integrated Development Environment + IDLE ==== .. moduleauthor:: Guido van Rossum -.. index:: - single: IDLE - single: Python Editor - single: Integrated Development Environment - IDLE is the Python IDE built with the :mod:`tkinter` GUI toolkit. IDLE has the following features: @@ -51,16 +51,16 @@ Recent Files Open a list of recent files +.. index:: + single: Class browser + single: Path browser + Class browser Show classes and methods in current file Path browser Show sys.path directories, modules, classes and methods -.. index:: - single: Class browser - single: Path browser - Save Save current window to the associated file (unsaved windows have a \* before and after the window title) @@ -137,7 +137,7 @@ Format menu (Editor window only) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Indent region Shift selected lines right by the indent width (default 4 spaces) @@ -263,6 +263,14 @@ Additional help sources may be added here with the Configure IDLE dialog under the General tab. +.. index:: + single: Cut + single: Copy + single: Paste + single: Set Breakpoint + single: Clear Breakpoint + single: breakpoints + Editor Window context menu ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -283,15 +291,6 @@ Clear Breakpoint Clears the breakpoint on that line. -.. index:: - single: Cut - single: Copy - single: Paste - single: Set Breakpoint - single: Clear Breakpoint - single: breakpoints - - Shell Window context menu ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -313,6 +312,9 @@ Editing and navigation ---------------------- +In this section, 'C' refers to the Control key on Windows and Unix and +the Command key on Mac OSX. + * :kbd:`Backspace` deletes to the left; :kbd:`Del` deletes to the right * :kbd:`C-Backspace` delete word left; :kbd:`C-Del` delete word to the right @@ -414,7 +416,6 @@ * :kbd:`C-c` interrupts executing command * :kbd:`C-d` sends end-of-file; closes window if typed at a ``>>>`` prompt - (this is :kbd:`C-z` on Windows). * :kbd:`Alt-/` (Expand word) is also useful to reduce typing @@ -507,6 +508,22 @@ name is '-', no script is executed but an interactive Python session is started; the arguments are still available in ``sys.argv``. +Running without a subprocess +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If IDLE is started with the -n command line switch it will run in a +single process and will not create the subprocess which runs the RPC +Python execution server. This can be useful if Python cannot create +the subprocess or the RPC socket interface on your platform. However, +in this mode user code is not isolated from IDLE itself. Also, the +environment is not restarted when Run/Run Module (F5) is selected. If +your code has been modified, you must reload() the affected modules and +re-import any specific items (e.g. from foo import baz) if the changes +are to take effect. For these reasons, it is preferable to run IDLE +with the default subprocess if at all possible. + +.. deprecated:: 3.4 + Additional help sources ----------------------- @@ -527,6 +544,7 @@ addition a user can create a custom key set in the Configure IDLE dialog under the keys tab. + Extensions ---------- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 06:56:11 2014 From: python-checkins at python.org (terry.reedy) Date: Thu, 04 Dec 2014 05:56:11 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2ODkz?= =?utf-8?q?=3A_Update_2=2E7_version_of_Idle_doc_to_match_3=2E4_doc_as_of_t?= =?utf-8?q?he_just?= Message-ID: <20141204055608.9549.30316@psf.io> https://hg.python.org/cpython/rev/feec1ea55127 changeset: 93717:feec1ea55127 branch: 2.7 parent: 93712:7af5d5493497 user: Terry Jan Reedy date: Thu Dec 04 00:55:46 2014 -0500 summary: Issue #16893: Update 2.7 version of Idle doc to match 3.4 doc as of the just committed 6db65ff985b6. files: Doc/library/idle.rst | 395 ++++++++++++++++++++++++------ 1 files changed, 307 insertions(+), 88 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -1,85 +1,97 @@ .. _idle: +.. index:: + single: IDLE + single: Python Editor + single: Integrated Development Environment + IDLE ==== .. moduleauthor:: Guido van Rossum -.. index:: - single: IDLE - single: Python Editor - single: Integrated Development Environment - IDLE is the Python IDE built with the :mod:`tkinter` GUI toolkit. IDLE has the following features: * coded in 100% pure Python, using the :mod:`tkinter` GUI toolkit -* cross-platform: works on Windows and Unix +* cross-platform: works on Windows, Unix, and Mac OS X -* multi-window text editor with multiple undo, Python colorizing and many other - features, e.g. smart indent and call tips +* multi-window text editor with multiple undo, Python colorizing, + smart indent, call tips, and many other features * Python shell window (a.k.a. interactive interpreter) -* debugger (not complete, but you can set breakpoints, view and step) +* debugger (not complete, but you can set breakpoints, view and step) Menus ----- +IDLE has two window types, the Shell window and the Editor window. It is +possible to have multiple editor windows simultaneously. IDLE's +menus dynamically change based on which window is currently selected. Each menu +documented below indicates which window type it is associated with. Click on +the dotted line at the top of a menu to "tear it off": a separate window +containing the menu is created (for Unix and Windows only). -File menu -^^^^^^^^^ +File menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ New file - create a new file editing window + Create a new file editing window Open... - open an existing file + Open an existing file Open module... - open an existing module (searches sys.path) + Open an existing module (searches sys.path) -Class browser - show classes and methods in current file - -Path browser - show sys.path directories, modules, classes and methods +Recent Files + Open a list of recent files .. index:: single: Class browser single: Path browser +Class browser + Show classes and methods in current file + +Path browser + Show sys.path directories, modules, classes and methods + Save - save current window to the associated file (unsaved windows have a \* before and - after the window title) + Save current window to the associated file (unsaved windows have a + \* before and after the window title) Save As... - save current window to new file, which becomes the associated file + Save current window to new file, which becomes the associated file Save Copy As... - save current window to different file without changing the associated file + Save current window to different file without changing the associated file + +Print Window + Print the current window Close - close current window (asks to save if unsaved) + Close current window (asks to save if unsaved) Exit - close all windows and quit IDLE (asks to save if unsaved) + Close all windows and quit IDLE (asks to save if unsaved) -Edit menu -^^^^^^^^^ +Edit menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Undo - Undo last change to current window (max 1000 changes) + Undo last change to current window (a maximum of 1000 changes may be undone) Redo Redo last undone change to current window Cut - Copy selection into system-wide clipboard; then delete selection + Copy selection into system-wide clipboard; then delete the selection Copy Copy selection into system-wide clipboard @@ -108,11 +120,30 @@ Go to line Ask for a line number and show that line +Expand word + Expand the word you have typed to match another word in the same buffer; + repeat to get a different expansion + +Show call tip + After an unclosed parenthesis for a function, open a small window with + function parameter hints + +Show surrounding parens + Highlight the surrounding parenthesis + +Show Completions + Open a scroll window allowing selection keywords and attributes. See + Completions below. + + +Format menu (Editor window only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Indent region - Shift selected lines right 4 spaces + Shift selected lines right by the indent width (default 4 spaces) Dedent region - Shift selected lines left 4 spaces + Shift selected lines left by the indent width (default 4 spaces) Comment out region Insert ## in front of selected lines @@ -121,67 +152,129 @@ Remove leading # or ## from selected lines Tabify region - Turns *leading* stretches of spaces into tabs + Turns *leading* stretches of spaces into tabs. (Note: We recommend using + 4 space blocks to indent Python code.) Untabify region - Turn *all* tabs into the right number of spaces + Turn *all* tabs into the correct number of spaces -Expand word - Expand the word you have typed to match another word in the same buffer; repeat - to get a different expansion +Toggle tabs + Open a dialog to switch between indenting with spaces and tabs. + +New Indent Width + Open a dialog to change indent width. The accepted default by the Python + community is 4 spaces. Format Paragraph - Reformat the current blank-line-separated paragraph + Reformat the current blank-line-separated paragraph. All lines in the + paragraph will be formatted to less than 80 columns. -Import module - Import or reload the current module - -Run script - Execute the current file in the __main__ namespace +Strip trailing whitespace + Removes any space characters after the end of the last non-space character .. index:: single: Import module single: Run script -Windows menu -^^^^^^^^^^^^ +Run menu (Editor window only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Zoom Height - toggles the window between normal size (24x80) and maximum height. +Python Shell + Open or wake up the Python Shell window -The rest of this menu lists the names of all open windows; select one to bring -it to the foreground (deiconifying it if necessary). +Check module + Check the syntax of the module currently open in the Editor window. If the + module has not been saved IDLE will prompt the user to save the code. +Run module + Restart the shell to clean the environment, then execute the currently + open module. If the module has not been saved IDLE will prompt the user + to save the code. -Debug menu -^^^^^^^^^^ +Shell menu (Shell window only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* in the Python Shell window only +View Last Restart + Scroll the shell window to the last Shell restart + +Restart Shell + Restart the shell to clean the environment + +Debug menu (Shell window only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Go to file/line Look around the insert point for a filename and line number, open the file, and show the line. Useful to view the source lines referenced in an - exception traceback. + exception traceback. Available in the context menu of the Shell window. -Debugger - Run commands in the shell under the debugger. +Debugger (toggle) + This feature is not complete and considered experimental. Run commands in + the shell under the debugger Stack viewer - Show the stack traceback of the last exception. + Show the stack traceback of the last exception Auto-open Stack Viewer - Open stack viewer on traceback. + Toggle automatically opening the stack viewer on unhandled exception .. index:: single: stack viewer single: debugger +Options menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Edit context menu -^^^^^^^^^^^^^^^^^ +Configure IDLE + Open a configuration dialog. Fonts, indentation, keybindings, and color + themes may be altered. Startup Preferences may be set, and additional + help sources can be specified. -* Right-click in Edit window (Control-click on OS X) +Code Context (toggle)(Editor Window only) + Open a pane at the top of the edit window which shows the block context + of the section of code which is scrolling off the top of the window. + +Windows menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Zoom Height + Toggles the window between normal size (40x80 initial setting) and maximum + height. The initial size is in the Configure IDLE dialog under the general + tab. + +The rest of this menu lists the names of all open windows; select one to bring +it to the foreground (deiconifying it if necessary). + +Help menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +About IDLE + Version, copyright, license, credits + +IDLE Help + Display a help file for IDLE detailing the menu options, basic editing and + navigation, and other tips. + +Python Docs + Access local Python documentation, if installed. Or will start a web browser + and open docs.python.org showing the latest Python documentation. + +Additional help sources may be added here with the Configure IDLE dialog under +the General tab. + +.. index:: + single: Cut + single: Copy + single: Paste + single: Set Breakpoint + single: Clear Breakpoint + single: breakpoints + +Editor Window context menu +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* Right-click in Editor window (Control-click on OS X) Cut Copy selection into system-wide clipboard; then delete selection @@ -198,17 +291,8 @@ Clear Breakpoint Clears the breakpoint on that line. -.. index:: - single: Cut - single: Copy - single: Paste - single: Set Breakpoint - single: Clear Breakpoint - single: breakpoints - - -Shell context menu -^^^^^^^^^^^^^^^^^^ +Shell Window context menu +^^^^^^^^^^^^^^^^^^^^^^^^^ * Right-click in Python Shell window (Control-click on OS X) @@ -225,19 +309,47 @@ Same as in Debug menu. -Basic editing and navigation ----------------------------- +Editing and navigation +---------------------- + +In this section, 'C' refers to the Control key on Windows and Unix and +the Command key on Mac OSX. * :kbd:`Backspace` deletes to the left; :kbd:`Del` deletes to the right +* :kbd:`C-Backspace` delete word left; :kbd:`C-Del` delete word to the right + * Arrow keys and :kbd:`Page Up`/:kbd:`Page Down` to move around +* :kbd:`C-LeftArrow` and :kbd:`C-RightArrow` moves by words + * :kbd:`Home`/:kbd:`End` go to begin/end of line * :kbd:`C-Home`/:kbd:`C-End` go to begin/end of file -* Some :program:`Emacs` bindings may also work, including :kbd:`C-B`, - :kbd:`C-P`, :kbd:`C-A`, :kbd:`C-E`, :kbd:`C-D`, :kbd:`C-L` +* Some useful Emacs bindings are inherited from Tcl/Tk: + + * :kbd:`C-a` beginning of line + + * :kbd:`C-e` end of line + + * :kbd:`C-k` kill line (but doesn't put it in clipboard) + + * :kbd:`C-l` center window around the insertion point + + * :kbd:`C-b` go backwards one character without deleting (usually you can + also use the cursor key for this) + + * :kbd:`C-f` go forward one character without deleting (usually you can + also use the cursor key for this) + + * :kbd:`C-p` go up one line (usually you can also use the cursor key for + this) + + * :kbd:`C-d` delete next character + +Standard keybindings (like :kbd:`C-c` to copy and :kbd:`C-v` to paste) +may work. Keybindings are selected in the Configure IDLE dialog. Automatic indentation @@ -246,27 +358,75 @@ After a block-opening statement, the next line is indented by 4 spaces (in the Python Shell window by one tab). After certain keywords (break, return etc.) the next line is dedented. In leading indentation, :kbd:`Backspace` deletes up -to 4 spaces if they are there. :kbd:`Tab` inserts 1-4 spaces (in the Python -Shell window one tab). See also the indent/dedent region commands in the edit -menu. +to 4 spaces if they are there. :kbd:`Tab` inserts spaces (in the Python +Shell window one tab), number depends on Indent width. Currently tabs +are restricted to four spaces due to Tcl/Tk limitations. +See also the indent/dedent region commands in the edit menu. + +Completions +^^^^^^^^^^^ + +Completions are supplied for functions, classes, and attributes of classes, +both built-in and user-defined. Completions are also provided for +filenames. + +The AutoCompleteWindow (ACW) will open after a predefined delay (default is +two seconds) after a '.' or (in a string) an os.sep is typed. If after one +of those characters (plus zero or more other characters) a tab is typed +the ACW will open immediately if a possible continuation is found. + +If there is only one possible completion for the characters entered, a +:kbd:`Tab` will supply that completion without opening the ACW. + +'Show Completions' will force open a completions window, by default the +:kbd:`C-space` will open a completions window. In an empty +string, this will contain the files in the current directory. On a +blank line, it will contain the built-in and user-defined functions and +classes in the current name spaces, plus any modules imported. If some +characters have been entered, the ACW will attempt to be more specific. + +If a string of characters is typed, the ACW selection will jump to the +entry most closely matching those characters. Entering a :kbd:`tab` will +cause the longest non-ambiguous match to be entered in the Editor window or +Shell. Two :kbd:`tab` in a row will supply the current ACW selection, as +will return or a double click. Cursor keys, Page Up/Down, mouse selection, +and the scroll wheel all operate on the ACW. + +"Hidden" attributes can be accessed by typing the beginning of hidden +name after a '.', e.g. '_'. This allows access to modules with +``__all__`` set, or to class-private attributes. + +Completions and the 'Expand Word' facility can save a lot of typing! + +Completions are currently limited to those in the namespaces. Names in +an Editor window which are not via ``__main__`` and :data:`sys.modules` will +not be found. Run the module once with your imports to correct this situation. +Note that IDLE itself places quite a few modules in sys.modules, so +much can be found by default, e.g. the re module. + +If you don't like the ACW popping up unbidden, simply make the delay +longer or disable the extension. Or another option is the delay could +be set to zero. Another alternative to preventing ACW popups is to +disable the call tips extension. Python Shell window ^^^^^^^^^^^^^^^^^^^ -* :kbd:`C-C` interrupts executing command +* :kbd:`C-c` interrupts executing command -* :kbd:`C-D` sends end-of-file; closes window if typed at a ``>>>`` prompt +* :kbd:`C-d` sends end-of-file; closes window if typed at a ``>>>`` prompt -* :kbd:`Alt-p` retrieves previous command matching what you have typed +* :kbd:`Alt-/` (Expand word) is also useful to reduce typing -* :kbd:`Alt-n` retrieves next + Command history -* :kbd:`Return` while on any previous command retrieves that command + * :kbd:`Alt-p` retrieves previous command matching what you have typed. On + OS X use :kbd:`C-p`. -* :kbd:`Alt-/` (Expand word) is also useful here + * :kbd:`Alt-n` retrieves next. On OS X use :kbd:`C-n`. -.. index:: single: indentation + * :kbd:`Return` while on any previous command retrieves that command Syntax colors @@ -308,17 +468,17 @@ Upon startup with the ``-s`` option, IDLE will execute the file referenced by the environment variables :envvar:`IDLESTARTUP` or :envvar:`PYTHONSTARTUP`. -Idle first checks for ``IDLESTARTUP``; if ``IDLESTARTUP`` is present the file -referenced is run. If ``IDLESTARTUP`` is not present, Idle checks for +IDLE first checks for ``IDLESTARTUP``; if ``IDLESTARTUP`` is present the file +referenced is run. If ``IDLESTARTUP`` is not present, IDLE checks for ``PYTHONSTARTUP``. Files referenced by these environment variables are -convenient places to store functions that are used frequently from the Idle +convenient places to store functions that are used frequently from the IDLE shell, or for executing import statements to import common modules. In addition, ``Tk`` also loads a startup file if it is present. Note that the Tk file is loaded unconditionally. This additional file is ``.Idle.py`` and is looked for in the user's home directory. Statements in this file will be executed in the Tk namespace, so this file is not useful for importing functions -to be used from Idle's Python shell. +to be used from IDLE's Python shell. Command line usage @@ -348,4 +508,63 @@ name is '-', no script is executed but an interactive Python session is started; the arguments are still available in ``sys.argv``. +Running without a subprocess +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +If IDLE is started with the -n command line switch it will run in a +single process and will not create the subprocess which runs the RPC +Python execution server. This can be useful if Python cannot create +the subprocess or the RPC socket interface on your platform. However, +in this mode user code is not isolated from IDLE itself. Also, the +environment is not restarted when Run/Run Module (F5) is selected. If +your code has been modified, you must reload() the affected modules and +re-import any specific items (e.g. from foo import baz) if the changes +are to take effect. For these reasons, it is preferable to run IDLE +with the default subprocess if at all possible. + +.. deprecated:: 3.4 + + +Additional help sources +----------------------- + +IDLE includes a help menu entry called "Python Docs" that will open the +extensive sources of help, including tutorials, available at docs.python.org. +Selected URLs can be added or removed from the help menu at any time using the +Configure IDLE dialog. See the IDLE help option in the help menu of IDLE for +more information. + + +Other preferences +----------------- + +The font preferences, highlighting, keys, and general preferences can be +changed via the Configure IDLE menu option. Be sure to note that +keys can be user defined, IDLE ships with four built in key sets. In +addition a user can create a custom key set in the Configure IDLE dialog +under the keys tab. + + +Extensions +---------- + +IDLE contains an extension facility. See the beginning of +config-extensions.def in the idlelib directory for further information. The +default extensions are currently: + +* FormatParagraph + +* AutoExpand + +* ZoomHeight + +* ScriptBinding + +* CallTips + +* ParenMatch + +* AutoComplete + +* CodeContext + -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 07:26:56 2014 From: python-checkins at python.org (terry.reedy) Date: Thu, 04 Dec 2014 06:26:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzMwNjg6?= =?utf-8?q?_Document_the_new_Configure_Extensions_dialog_and_menu_entry=2E?= Message-ID: <20141204062656.95982.88129@psf.io> https://hg.python.org/cpython/rev/395673aac686 changeset: 93718:395673aac686 branch: 2.7 user: Terry Jan Reedy date: Thu Dec 04 01:22:41 2014 -0500 summary: Issue #3068: Document the new Configure Extensions dialog and menu entry. files: Doc/library/idle.rst | 30 +++++++++++++++++++----------- 1 files changed, 19 insertions(+), 11 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -231,6 +231,10 @@ themes may be altered. Startup Preferences may be set, and additional help sources can be specified. +Configure Extensions + Open a configuration dialog for setting preferences for extensions + (discussed below). + Code Context (toggle)(Editor Window only) Open a pane at the top of the edit window which shows the block context of the section of code which is scrolling off the top of the window. @@ -525,8 +529,11 @@ .. deprecated:: 3.4 +Help and preferences +-------------------- + Additional help sources ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ IDLE includes a help menu entry called "Python Docs" that will open the extensive sources of help, including tutorials, available at docs.python.org. @@ -535,22 +542,22 @@ more information. -Other preferences ------------------ +Setting preferences +^^^^^^^^^^^^^^^^^^^ The font preferences, highlighting, keys, and general preferences can be -changed via the Configure IDLE menu option. Be sure to note that -keys can be user defined, IDLE ships with four built in key sets. In -addition a user can create a custom key set in the Configure IDLE dialog -under the keys tab. +changed via Configure IDLE on the Option menu. Keys can be user defined; +IDLE ships with four built in key sets. In addition a user can create a +custom key set in the Configure IDLE dialog under the keys tab. Extensions ----------- +^^^^^^^^^^ -IDLE contains an extension facility. See the beginning of -config-extensions.def in the idlelib directory for further information. The -default extensions are currently: +IDLE contains an extension facility. Peferences for extensions can be +changed with Configure Extensions. See the beginning of config-extensions.def +in the idlelib directory for further information. The default extensions +are currently: * FormatParagraph @@ -568,3 +575,4 @@ * CodeContext +* RstripExtension -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 07:26:56 2014 From: python-checkins at python.org (terry.reedy) Date: Thu, 04 Dec 2014 06:26:56 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzMwNjg6?= =?utf-8?q?_Document_the_new_Configure_Extensions_dialog_and_menu_entry=2E?= Message-ID: <20141204062656.46361.95377@psf.io> https://hg.python.org/cpython/rev/b099cc290ae9 changeset: 93719:b099cc290ae9 branch: 3.4 parent: 93715:6db65ff985b6 user: Terry Jan Reedy date: Thu Dec 04 01:26:04 2014 -0500 summary: Issue #3068: Document the new Configure Extensions dialog and menu entry. files: Doc/library/idle.rst | 30 +++++++++++++++++++----------- 1 files changed, 19 insertions(+), 11 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -231,6 +231,10 @@ themes may be altered. Startup Preferences may be set, and additional help sources can be specified. +Configure Extensions + Open a configuration dialog for setting preferences for extensions + (discussed below). + Code Context (toggle)(Editor Window only) Open a pane at the top of the edit window which shows the block context of the section of code which is scrolling off the top of the window. @@ -525,8 +529,11 @@ .. deprecated:: 3.4 +Help and preferences +-------------------- + Additional help sources ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ IDLE includes a help menu entry called "Python Docs" that will open the extensive sources of help, including tutorials, available at docs.python.org. @@ -535,22 +542,22 @@ more information. -Other preferences ------------------ +Setting preferences +^^^^^^^^^^^^^^^^^^^ The font preferences, highlighting, keys, and general preferences can be -changed via the Configure IDLE menu option. Be sure to note that -keys can be user defined, IDLE ships with four built in key sets. In -addition a user can create a custom key set in the Configure IDLE dialog -under the keys tab. +changed via Configure IDLE on the Option menu. Keys can be user defined; +IDLE ships with four built in key sets. In addition a user can create a +custom key set in the Configure IDLE dialog under the keys tab. Extensions ----------- +^^^^^^^^^^ -IDLE contains an extension facility. See the beginning of -config-extensions.def in the idlelib directory for further information. The -default extensions are currently: +IDLE contains an extension facility. Peferences for extensions can be +changed with Configure Extensions. See the beginning of config-extensions.def +in the idlelib directory for further information. The default extensions +are currently: * FormatParagraph @@ -568,3 +575,4 @@ * CodeContext +* RstripExtension -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 07:26:57 2014 From: python-checkins at python.org (terry.reedy) Date: Thu, 04 Dec 2014 06:26:57 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <20141204062656.31245.81650@psf.io> https://hg.python.org/cpython/rev/cd282dd0cfe8 changeset: 93720:cd282dd0cfe8 parent: 93716:dc146e018fe3 parent: 93719:b099cc290ae9 user: Terry Jan Reedy date: Thu Dec 04 01:26:35 2014 -0500 summary: Merge with 3.4 files: Doc/library/idle.rst | 30 +++++++++++++++++++----------- 1 files changed, 19 insertions(+), 11 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -231,6 +231,10 @@ themes may be altered. Startup Preferences may be set, and additional help sources can be specified. +Configure Extensions + Open a configuration dialog for setting preferences for extensions + (discussed below). + Code Context (toggle)(Editor Window only) Open a pane at the top of the edit window which shows the block context of the section of code which is scrolling off the top of the window. @@ -525,8 +529,11 @@ .. deprecated:: 3.4 +Help and preferences +-------------------- + Additional help sources ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ IDLE includes a help menu entry called "Python Docs" that will open the extensive sources of help, including tutorials, available at docs.python.org. @@ -535,22 +542,22 @@ more information. -Other preferences ------------------ +Setting preferences +^^^^^^^^^^^^^^^^^^^ The font preferences, highlighting, keys, and general preferences can be -changed via the Configure IDLE menu option. Be sure to note that -keys can be user defined, IDLE ships with four built in key sets. In -addition a user can create a custom key set in the Configure IDLE dialog -under the keys tab. +changed via Configure IDLE on the Option menu. Keys can be user defined; +IDLE ships with four built in key sets. In addition a user can create a +custom key set in the Configure IDLE dialog under the keys tab. Extensions ----------- +^^^^^^^^^^ -IDLE contains an extension facility. See the beginning of -config-extensions.def in the idlelib directory for further information. The -default extensions are currently: +IDLE contains an extension facility. Peferences for extensions can be +changed with Configure Extensions. See the beginning of config-extensions.def +in the idlelib directory for further information. The default extensions +are currently: * FormatParagraph @@ -568,3 +575,4 @@ * CodeContext +* RstripExtension -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Thu Dec 4 10:29:27 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 04 Dec 2014 10:29:27 +0100 Subject: [Python-checkins] Daily reference leaks (7509a0607c40): sum=3 Message-ID: results for 7509a0607c40 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogI5yidF', '-x'] From tjreedy at udel.edu Thu Dec 4 17:16:44 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 04 Dec 2014 11:16:44 -0500 Subject: [Python-checkins] Daily reference leaks (7509a0607c40): sum=3 In-Reply-To: References: Message-ID: <548088EC.2070308@udel.edu> On 12/4/2014 4:29 AM, solipsis at pitrou.net wrote: > results for 7509a0607c40 on branch "default" > -------------------------------------------- >> test_functools leaked [0, 0, 3] memory blocks, sum=3 > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogI5yidF', '-x'] I tried to see if I could at least find the offending test, but if I am reading the below correctly, there is no leak on Windows. F:\Python\dev\35\PCbuild>python_d -m test -uall -R 3:3 test_functools [1/1] test_functools beginning 6 repetitions 123456 ...... 1 test OK. both with 2-week-old build and fresh build tjr From python-checkins at python.org Thu Dec 4 23:09:42 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 04 Dec 2014 22:09:42 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNjg1?= =?utf-8?q?=3A_Fix_test=5Fpause=5Freading=28=29_of_asyncio_test=5Fsubproce?= =?utf-8?q?ss?= Message-ID: <20141204220939.76714.13759@psf.io> https://hg.python.org/cpython/rev/2c18dd99829c changeset: 93722:2c18dd99829c branch: 3.4 user: Victor Stinner date: Thu Dec 04 23:06:13 2014 +0100 summary: Issue #22685: Fix test_pause_reading() of asyncio test_subprocess * mock also resume_reading() * ensure that resume_reading() is called files: Lib/test/test_asyncio/test_subprocess.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -181,6 +181,7 @@ loop=self.loop) stdout_transport = proc._transport.get_pipe_transport(1) stdout_transport.pause_reading = mock.Mock() + stdout_transport.resume_reading = mock.Mock() stdout, stderr = yield from proc.communicate() @@ -195,6 +196,7 @@ self.assertEqual(stdout, b'x' * size) self.assertTrue(transport.pause_reading.called) + self.assertTrue(transport.resume_reading.called) if sys.platform != 'win32': -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 23:09:42 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 04 Dec 2014 22:09:42 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogSW5p?= =?utf-8?q?tialize_more_Future_and_Task_attributes_in_the_class_definition?= =?utf-8?q?_to?= Message-ID: <20141204220937.21256.72378@psf.io> https://hg.python.org/cpython/rev/4041380e6a2b changeset: 93721:4041380e6a2b branch: 3.4 parent: 93719:b099cc290ae9 user: Victor Stinner date: Thu Dec 04 23:00:13 2014 +0100 summary: asyncio: Initialize more Future and Task attributes in the class definition to avoid attribute errors in destructors. files: Lib/asyncio/futures.py | 3 +-- Lib/asyncio/tasks.py | 7 ++++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -135,6 +135,7 @@ _result = None _exception = None _loop = None + _source_traceback = None _blocking = False # proper use of future (yield vs yield from) @@ -155,8 +156,6 @@ self._callbacks = [] if self._loop.get_debug(): self._source_traceback = traceback.extract_stack(sys._getframe(1)) - else: - self._source_traceback = None def _format_callbacks(self): cb = self._callbacks diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -41,6 +41,10 @@ # all running event loops. {EventLoop: Task} _current_tasks = {} + # If False, don't log a message if the task is destroyed whereas its + # status is still pending + _log_destroy_pending = True + @classmethod def current_task(cls, loop=None): """Return the currently running task in an event loop or None. @@ -73,9 +77,6 @@ self._must_cancel = False self._loop.call_soon(self._step) self.__class__._all_tasks.add(self) - # If False, don't log a message if the task is destroyed whereas its - # status is still pending - self._log_destroy_pending = True # On Python 3.3 or older, objects with a destructor that are part of a # reference cycle are never destroyed. That's not the case any more on -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 23:09:43 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 04 Dec 2014 22:09:43 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMjky?= =?utf-8?q?2=3A_More_EventLoop_methods_fail_if_the_loop_is_closed=2E_Initi?= =?utf-8?q?al_patch?= Message-ID: <20141204220940.21252.40183@psf.io> https://hg.python.org/cpython/rev/226f870b387d changeset: 93723:226f870b387d branch: 3.4 user: Victor Stinner date: Thu Dec 04 23:07:47 2014 +0100 summary: Closes #22922: More EventLoop methods fail if the loop is closed. Initial patch written by Torsten Landschoff. create_task(), call_at(), call_soon(), call_soon_threadsafe() and run_in_executor() now raise an error if the event loop is closed. files: Lib/asyncio/base_events.py | 4 + Lib/asyncio/unix_events.py | 1 + Lib/test/test_asyncio/test_events.py | 35 +++++++++++++++- 3 files changed, 39 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -177,6 +177,7 @@ Return a task object. """ + self._check_closed() task = tasks.Task(coro, loop=self) if task._source_traceback: del task._source_traceback[-1] @@ -360,6 +361,7 @@ if (coroutines.iscoroutine(callback) or coroutines.iscoroutinefunction(callback)): raise TypeError("coroutines cannot be used with call_at()") + self._check_closed() if self._debug: self._assert_is_current_event_loop() timer = events.TimerHandle(when, callback, args, self) @@ -390,6 +392,7 @@ raise TypeError("coroutines cannot be used with call_soon()") if self._debug and check_loop: self._assert_is_current_event_loop() + self._check_closed() handle = events.Handle(callback, args, self) if handle._source_traceback: del handle._source_traceback[-1] @@ -426,6 +429,7 @@ if (coroutines.iscoroutine(callback) or coroutines.iscoroutinefunction(callback)): raise TypeError("coroutines cannot be used with run_in_executor()") + self._check_closed() if isinstance(callback, events.Handle): assert not args assert not isinstance(callback, events.TimerHandle) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -71,6 +71,7 @@ or coroutines.iscoroutinefunction(callback)): raise TypeError("coroutines cannot be used with add_signal_handler()") self._check_signal(sig) + self._check_closed() try: # set_wakeup_fd() raises ValueError if this is not the # main thread. By calling it early we ensure that an diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -226,7 +226,8 @@ def tearDown(self): # just in case if we have transport close callbacks - test_utils.run_briefly(self.loop) + if not self.loop.is_closed(): + test_utils.run_briefly(self.loop) self.loop.close() gc.collect() @@ -1434,6 +1435,38 @@ with self.assertRaises(RuntimeError): self.loop.run_until_complete(coro) + def test_close(self): + self.loop.close() + + @asyncio.coroutine + def test(): + pass + + func = lambda: False + coro = test() + self.addCleanup(coro.close) + + # operation blocked when the loop is closed + with self.assertRaises(RuntimeError): + self.loop.run_forever() + with self.assertRaises(RuntimeError): + fut = asyncio.Future(loop=self.loop) + self.loop.run_until_complete(fut) + with self.assertRaises(RuntimeError): + self.loop.call_soon(func) + with self.assertRaises(RuntimeError): + self.loop.call_soon_threadsafe(func) + with self.assertRaises(RuntimeError): + self.loop.call_later(1.0, func) + with self.assertRaises(RuntimeError): + self.loop.call_at(self.loop.time() + .0, func) + with self.assertRaises(RuntimeError): + self.loop.run_in_executor(None, func) + with self.assertRaises(RuntimeError): + self.loop.create_task(coro) + with self.assertRaises(RuntimeError): + self.loop.add_signal_handler(signal.SIGTERM, func) + class SubprocessTestsMixin: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Thu Dec 4 23:09:45 2014 From: python-checkins at python.org (victor.stinner) Date: Thu, 04 Dec 2014 22:09:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20141204220942.27499.63560@psf.io> https://hg.python.org/cpython/rev/eec6ec024a35 changeset: 93724:eec6ec024a35 parent: 93720:cd282dd0cfe8 parent: 93723:226f870b387d user: Victor Stinner date: Thu Dec 04 23:08:37 2014 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 4 + Lib/asyncio/futures.py | 3 +- Lib/asyncio/tasks.py | 7 +- Lib/asyncio/unix_events.py | 1 + Lib/test/test_asyncio/test_events.py | 35 +++++++++++++++- 5 files changed, 44 insertions(+), 6 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -177,6 +177,7 @@ Return a task object. """ + self._check_closed() task = tasks.Task(coro, loop=self) if task._source_traceback: del task._source_traceback[-1] @@ -360,6 +361,7 @@ if (coroutines.iscoroutine(callback) or coroutines.iscoroutinefunction(callback)): raise TypeError("coroutines cannot be used with call_at()") + self._check_closed() if self._debug: self._assert_is_current_event_loop() timer = events.TimerHandle(when, callback, args, self) @@ -390,6 +392,7 @@ raise TypeError("coroutines cannot be used with call_soon()") if self._debug and check_loop: self._assert_is_current_event_loop() + self._check_closed() handle = events.Handle(callback, args, self) if handle._source_traceback: del handle._source_traceback[-1] @@ -426,6 +429,7 @@ if (coroutines.iscoroutine(callback) or coroutines.iscoroutinefunction(callback)): raise TypeError("coroutines cannot be used with run_in_executor()") + self._check_closed() if isinstance(callback, events.Handle): assert not args assert not isinstance(callback, events.TimerHandle) diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -135,6 +135,7 @@ _result = None _exception = None _loop = None + _source_traceback = None _blocking = False # proper use of future (yield vs yield from) @@ -155,8 +156,6 @@ self._callbacks = [] if self._loop.get_debug(): self._source_traceback = traceback.extract_stack(sys._getframe(1)) - else: - self._source_traceback = None def _format_callbacks(self): cb = self._callbacks diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -41,6 +41,10 @@ # all running event loops. {EventLoop: Task} _current_tasks = {} + # If False, don't log a message if the task is destroyed whereas its + # status is still pending + _log_destroy_pending = True + @classmethod def current_task(cls, loop=None): """Return the currently running task in an event loop or None. @@ -73,9 +77,6 @@ self._must_cancel = False self._loop.call_soon(self._step) self.__class__._all_tasks.add(self) - # If False, don't log a message if the task is destroyed whereas its - # status is still pending - self._log_destroy_pending = True # On Python 3.3 or older, objects with a destructor that are part of a # reference cycle are never destroyed. That's not the case any more on diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -71,6 +71,7 @@ or coroutines.iscoroutinefunction(callback)): raise TypeError("coroutines cannot be used with add_signal_handler()") self._check_signal(sig) + self._check_closed() try: # set_wakeup_fd() raises ValueError if this is not the # main thread. By calling it early we ensure that an diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -226,7 +226,8 @@ def tearDown(self): # just in case if we have transport close callbacks - test_utils.run_briefly(self.loop) + if not self.loop.is_closed(): + test_utils.run_briefly(self.loop) self.loop.close() gc.collect() @@ -1434,6 +1435,38 @@ with self.assertRaises(RuntimeError): self.loop.run_until_complete(coro) + def test_close(self): + self.loop.close() + + @asyncio.coroutine + def test(): + pass + + func = lambda: False + coro = test() + self.addCleanup(coro.close) + + # operation blocked when the loop is closed + with self.assertRaises(RuntimeError): + self.loop.run_forever() + with self.assertRaises(RuntimeError): + fut = asyncio.Future(loop=self.loop) + self.loop.run_until_complete(fut) + with self.assertRaises(RuntimeError): + self.loop.call_soon(func) + with self.assertRaises(RuntimeError): + self.loop.call_soon_threadsafe(func) + with self.assertRaises(RuntimeError): + self.loop.call_later(1.0, func) + with self.assertRaises(RuntimeError): + self.loop.call_at(self.loop.time() + .0, func) + with self.assertRaises(RuntimeError): + self.loop.run_in_executor(None, func) + with self.assertRaises(RuntimeError): + self.loop.create_task(coro) + with self.assertRaises(RuntimeError): + self.loop.add_signal_handler(signal.SIGTERM, func) + class SubprocessTestsMixin: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 01:27:51 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 05 Dec 2014 00:27:51 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322922=3A_Fix_Proa?= =?utf-8?q?ctorEventLoop=2Eclose=28=29?= Message-ID: <20141205002742.74613.95371@psf.io> https://hg.python.org/cpython/rev/6a955db1f78f changeset: 93725:6a955db1f78f user: Victor Stinner date: Fri Dec 05 01:25:21 2014 +0100 summary: Issue #22922: Fix ProactorEventLoop.close() Call _stop_accept_futures() before sestting the _closed attribute, otherwise call_soon() raises an error. files: Lib/asyncio/proactor_events.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -387,11 +387,13 @@ sock, protocol, waiter, extra) def close(self): + if self._running: + raise RuntimeError("Cannot close a running event loop") if self.is_closed(): return - super().close() self._stop_accept_futures() self._close_self_pipe() + super().close() self._proactor.close() self._proactor = None self._selector = None -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 01:45:37 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 05 Dec 2014 00:45:37 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyOTIy?= =?utf-8?q?=3A_Fix_ProactorEventLoop=2Eclose=28=29?= Message-ID: <20141205004531.27517.5262@psf.io> https://hg.python.org/cpython/rev/0a7d956a2f2d changeset: 93726:0a7d956a2f2d branch: 3.4 parent: 93723:226f870b387d user: Victor Stinner date: Fri Dec 05 01:43:42 2014 +0100 summary: Issue #22922: Fix ProactorEventLoop.close() Call _stop_accept_futures() before sestting the _closed attribute, otherwise call_soon() raises an error. files: Lib/asyncio/proactor_events.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -387,11 +387,13 @@ sock, protocol, waiter, extra) def close(self): + if self._running: + raise RuntimeError("Cannot close a running event loop") if self.is_closed(): return - super().close() self._stop_accept_futures() self._close_self_pipe() + super().close() self._proactor.close() self._proactor = None self._selector = None -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 01:45:37 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 05 Dec 2014 00:45:37 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?= Message-ID: <20141205004532.74595.79979@psf.io> https://hg.python.org/cpython/rev/2fa3c68de07e changeset: 93728:2fa3c68de07e parent: 93725:6a955db1f78f parent: 93727:83bddbfbd3a4 user: Victor Stinner date: Fri Dec 05 01:44:31 2014 +0100 summary: Merge 3.4 (asyncio) files: Lib/asyncio/base_events.py | 14 +++++- Lib/test/test_asyncio/test_base_events.py | 25 +++++++++++ 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -102,6 +102,16 @@ raise _StopError +def _run_until_complete_cb(fut): + exc = fut._exception + if (isinstance(exc, BaseException) + and not isinstance(exc, Exception)): + # Issue #22429: run_forever() already finished, no need to + # stop it. + return + _raise_stop_error() + + class Server(events.AbstractServer): def __init__(self, loop, sockets): @@ -268,7 +278,7 @@ # is no need to log the "destroy pending task" message future._log_destroy_pending = False - future.add_done_callback(_raise_stop_error) + future.add_done_callback(_run_until_complete_cb) try: self.run_forever() except: @@ -278,7 +288,7 @@ # local task. future.exception() raise - future.remove_done_callback(_raise_stop_error) + future.remove_done_callback(_run_until_complete_cb) if not future.done(): raise RuntimeError('Event loop stopped before Future completed.') diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -638,6 +638,31 @@ self.assertFalse(self.loop.call_exception_handler.called) + def test_run_until_complete_baseexception(self): + # Python issue #22429: run_until_complete() must not schedule a pending + # call to stop() if the future raised a BaseException + @asyncio.coroutine + def raise_keyboard_interrupt(): + raise KeyboardInterrupt + + self.loop._process_events = mock.Mock() + + try: + self.loop.run_until_complete(raise_keyboard_interrupt()) + except KeyboardInterrupt: + pass + + def func(): + self.loop.stop() + func.called = True + func.called = False + try: + self.loop.call_soon(func) + self.loop.run_forever() + except KeyboardInterrupt: + pass + self.assertTrue(func.called) + class MyProto(asyncio.Protocol): done = None -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 01:45:37 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 05 Dec 2014 00:45:37 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogQ2xvc2VzICMyMjQy?= =?utf-8?q?9=2C_asyncio=3A_Fix_EventLoop=2Erun=5Funtil=5Fcomplete=28=29=2C?= =?utf-8?q?_don=27t_stop_the?= Message-ID: <20141205004531.21272.8112@psf.io> https://hg.python.org/cpython/rev/83bddbfbd3a4 changeset: 93727:83bddbfbd3a4 branch: 3.4 user: Victor Stinner date: Fri Dec 05 01:44:10 2014 +0100 summary: Closes #22429, asyncio: Fix EventLoop.run_until_complete(), don't stop the event loop if a BaseException is raised, because the event loop is already stopped. files: Lib/asyncio/base_events.py | 14 +++++- Lib/test/test_asyncio/test_base_events.py | 25 +++++++++++ 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -102,6 +102,16 @@ raise _StopError +def _run_until_complete_cb(fut): + exc = fut._exception + if (isinstance(exc, BaseException) + and not isinstance(exc, Exception)): + # Issue #22429: run_forever() already finished, no need to + # stop it. + return + _raise_stop_error() + + class Server(events.AbstractServer): def __init__(self, loop, sockets): @@ -268,7 +278,7 @@ # is no need to log the "destroy pending task" message future._log_destroy_pending = False - future.add_done_callback(_raise_stop_error) + future.add_done_callback(_run_until_complete_cb) try: self.run_forever() except: @@ -278,7 +288,7 @@ # local task. future.exception() raise - future.remove_done_callback(_raise_stop_error) + future.remove_done_callback(_run_until_complete_cb) if not future.done(): raise RuntimeError('Event loop stopped before Future completed.') diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -638,6 +638,31 @@ self.assertFalse(self.loop.call_exception_handler.called) + def test_run_until_complete_baseexception(self): + # Python issue #22429: run_until_complete() must not schedule a pending + # call to stop() if the future raised a BaseException + @asyncio.coroutine + def raise_keyboard_interrupt(): + raise KeyboardInterrupt + + self.loop._process_events = mock.Mock() + + try: + self.loop.run_until_complete(raise_keyboard_interrupt()) + except KeyboardInterrupt: + pass + + def func(): + self.loop.stop() + func.called = True + func.called = False + try: + self.loop.call_soon(func) + self.loop.run_forever() + except KeyboardInterrupt: + pass + self.assertTrue(func.called) + class MyProto(asyncio.Protocol): done = None -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 04:49:01 2014 From: python-checkins at python.org (yury.selivanov) Date: Fri, 05 Dec 2014 03:49:01 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogZG9jcy5pbnNwZWN0?= =?utf-8?q?=3A_Fix_BoundArguments_example=2E_Issue_=2322998=2E?= Message-ID: <20141205034901.21254.4912@psf.io> https://hg.python.org/cpython/rev/71c38c233e5c changeset: 93729:71c38c233e5c branch: 3.4 parent: 93727:83bddbfbd3a4 user: Yury Selivanov date: Thu Dec 04 22:47:44 2014 -0500 summary: docs.inspect: Fix BoundArguments example. Issue #22998. files: Doc/library/inspect.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -642,7 +642,8 @@ ((5,), {}) >>> for param in sig.parameters.values(): - ... if param.name not in ba.arguments: + ... if (param.name not in ba.arguments + ... and param.default is not param.empty): ... ba.arguments[param.name] = param.default >>> ba.args, ba.kwargs -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 04:49:02 2014 From: python-checkins at python.org (yury.selivanov) Date: Fri, 05 Dec 2014 03:49:02 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_docs=2Einspect=3A_Fix_Boun?= =?utf-8?q?dArguments_example=2E_Issue_=2322998=2E?= Message-ID: <20141205034901.74601.18400@psf.io> https://hg.python.org/cpython/rev/697adefaba6b changeset: 93730:697adefaba6b parent: 93728:2fa3c68de07e user: Yury Selivanov date: Thu Dec 04 22:48:47 2014 -0500 summary: docs.inspect: Fix BoundArguments example. Issue #22998. files: Doc/library/inspect.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -678,7 +678,8 @@ ((5,), {}) >>> for param in sig.parameters.values(): - ... if param.name not in ba.arguments: + ... if (param.name not in ba.arguments + ... and param.default is not param.empty): ... ba.arguments[param.name] = param.default >>> ba.args, ba.kwargs -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 05:29:26 2014 From: python-checkins at python.org (terry.reedy) Date: Fri, 05 Dec 2014 04:29:26 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_from_3=2E4_after_patch_separately_committed_as_697?= =?utf-8?q?adefaba6b=2E?= Message-ID: <20141205042925.27489.32064@psf.io> https://hg.python.org/cpython/rev/e968803baaa4 changeset: 93731:e968803baaa4 parent: 93730:697adefaba6b parent: 93729:71c38c233e5c user: Terry Jan Reedy date: Thu Dec 04 23:29:07 2014 -0500 summary: Merge from 3.4 after patch separately committed as 697adefaba6b. files: -- Repository URL: https://hg.python.org/cpython From tjreedy at udel.edu Fri Dec 5 05:34:18 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 04 Dec 2014 23:34:18 -0500 Subject: [Python-checkins] cpython: docs.inspect: Fix BoundArguments example. Issue #22998. In-Reply-To: <20141205034901.74601.18400@psf.io> References: <20141205034901.74601.18400@psf.io> Message-ID: <548135CA.9070702@udel.edu> On 12/4/2014 10:49 PM, yury.selivanov wrote: > https://hg.python.org/cpython/rev/697adefaba6b > changeset: 93730:697adefaba6b > parent: 93728:2fa3c68de07e > user: Yury Selivanov > date: Thu Dec 04 22:48:47 2014 -0500 > summary: > docs.inspect: Fix BoundArguments example. Issue #22998. > > files: > Doc/library/inspect.rst | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > > diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst > --- a/Doc/library/inspect.rst > +++ b/Doc/library/inspect.rst > @@ -678,7 +678,8 @@ > ((5,), {}) > > >>> for param in sig.parameters.values(): > - ... if param.name not in ba.arguments: > + ... if (param.name not in ba.arguments > + ... and param.default is not param.empty): > ... ba.arguments[param.name] = param.default > > >>> ba.args, ba.kwargs Yury, it appears that you patched 3.5 directly instead of merging the 3.4 commit 71c38c233e5c into 3.5. The result was that the 3.4 commit was left unmerged, potentially interfering with the next person doing a merge. I believe I fixed it with what seems effectively to be a null merge. Terry From python-checkins at python.org Fri Dec 5 08:43:49 2014 From: python-checkins at python.org (terry.reedy) Date: Fri, 05 Dec 2014 07:43:49 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Edit_Menus_sec?= =?utf-8?q?tion_of_Idle_doc=2E__Make_menu_entries_match_current_Idle_in_or?= =?utf-8?q?der?= Message-ID: <20141205074349.74609.6054@psf.io> https://hg.python.org/cpython/rev/c3c060e5ba99 changeset: 93733:c3c060e5ba99 branch: 3.4 parent: 93729:71c38c233e5c user: Terry Jan Reedy date: Fri Dec 05 02:43:07 2014 -0500 summary: Edit Menus section of Idle doc. Make menu entries match current Idle in order and case. Edit some of the explanatory sentences and end all with a period. files: Doc/library/idle.rst | 261 ++++++++++++++++-------------- 1 files changed, 138 insertions(+), 123 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -29,136 +29,147 @@ Menus ----- -IDLE has two window types, the Shell window and the Editor window. It is -possible to have multiple editor windows simultaneously. IDLE's -menus dynamically change based on which window is currently selected. Each menu -documented below indicates which window type it is associated with. Click on -the dotted line at the top of a menu to "tear it off": a separate window -containing the menu is created (for Unix and Windows only). +IDLE has two main window types, the Shell window and the Editor window. It is +possible to have multiple editor windows simultaneously. Output windows, such +as used for Edit / Find in Files, are a subtype of edit window. They currently +have the same top menu as Editor windows but a different default title and +context menu. + +IDLE's menus dynamically change based on which window is currently selected. +Each menu documented below indicates which window type it is associated with. +Click on the dotted line at the top of a menu to "tear it off": a separate +window containing the menu is created (for Unix and Windows only). File menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -New file - Create a new file editing window +New File + Create a new file editing window. Open... - Open an existing file - -Open module... - Open an existing module (searches sys.path) + Open an existing file with an Open dialog. Recent Files - Open a list of recent files + Open a list of recent files. Click one to open it. + +Open Module... + Open an existing module (searches sys.path). .. index:: single: Class browser single: Path browser -Class browser - Show classes and methods in current file +Class Browser + Show functions, classes, and methods in the current Editor file in a + tree structure. In the shell, open a module first. -Path browser - Show sys.path directories, modules, classes and methods +Path Browser + Show sys.path directories, modules, functions, classes and methods in a + tree structure. Save - Save current window to the associated file (unsaved windows have a - \* before and after the window title) + Save the current window to the associated file, if there is one. Windows + that have been changed since being opened or last saved have a \* before + and after the window title. If there is no associated file, + do Save As instead. Save As... - Save current window to new file, which becomes the associated file + Save the current window with a Save As dialog. The file saved becomes the + new associated file for the window. Save Copy As... - Save current window to different file without changing the associated file + Save the current window to different file without changing the associated + file. Print Window - Print the current window + Print the current window to the default printer. Close - Close current window (asks to save if unsaved) + Close the current window (ask to save if unsaved). Exit - Close all windows and quit IDLE (asks to save if unsaved) - + Close all windows and quit IDLE (ask to save unsaved windows). Edit menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Undo - Undo last change to current window (a maximum of 1000 changes may be undone) + Undo the last change to the current window. A maximum of 1000 changes may + be undone. Redo - Redo last undone change to current window + Redo the last undone change to the current window. Cut - Copy selection into system-wide clipboard; then delete the selection + Copy selection into the system-wide clipboard; then delete the selection. Copy - Copy selection into system-wide clipboard + Copy selection into the system-wide clipboard. Paste - Insert system-wide clipboard into window + Insert contents of the system-wide clipboard into the current window. + +The clipboard functions are also available in context menus. Select All - Select the entire contents of the edit buffer + Select the entire contents of the current window. Find... - Open a search dialog box with many options + Open a search dialog with many options -Find again - Repeat last search +Find Again + Repeat the last search, if there is one. -Find selection - Search for the string in the selection +Find Selection + Search for the currently selected string, if there is one. Find in Files... - Open a search dialog box for searching files + Open a file search dialog. Put results in an new output window. Replace... - Open a search-and-replace dialog box + Open a search-and-replace dialog. -Go to line - Ask for a line number and show that line +Go to Line + Move cursor to the line number requested and make that line visible. -Expand word - Expand the word you have typed to match another word in the same buffer; - repeat to get a different expansion +Show Completions + Open a scrollable list allowing selection of keywords and attributes. See + Completions in the Tips sections below. + +Expand Word + Expand a prefix you have typed to match a full word in the same window; + repeat to get a different expansion. Show call tip After an unclosed parenthesis for a function, open a small window with - function parameter hints + function parameter hints. Show surrounding parens - Highlight the surrounding parenthesis - -Show Completions - Open a scroll window allowing selection keywords and attributes. See - Completions below. - + Highlight the surrounding parenthesis. Format menu (Editor window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Indent region - Shift selected lines right by the indent width (default 4 spaces) +Indent Region + Shift selected lines right by the indent width (default 4 spaces). -Dedent region - Shift selected lines left by the indent width (default 4 spaces) +Dedent Region + Shift selected lines left by the indent width (default 4 spaces). -Comment out region - Insert ## in front of selected lines +Comment Out Region + Insert ## in front of selected lines. -Uncomment region - Remove leading # or ## from selected lines +Uncomment Region + Remove leading # or ## from selected lines. -Tabify region - Turns *leading* stretches of spaces into tabs. (Note: We recommend using +Tabify Region + Turn *leading* stretches of spaces into tabs. (Note: We recommend using 4 space blocks to indent Python code.) -Untabify region - Turn *all* tabs into the correct number of spaces +Untabify Region + Turn *all* tabs into the correct number of spaces. -Toggle tabs +Toggle Tabs Open a dialog to switch between indenting with spaces and tabs. New Indent Width @@ -166,62 +177,67 @@ community is 4 spaces. Format Paragraph - Reformat the current blank-line-separated paragraph. All lines in the - paragraph will be formatted to less than 80 columns. + Reformat the current blank-line-delimited paragraph in comment block or + multiline string or selected line in a string. All lines in the + paragraph will be formatted to less than N columns, where N defaults to 72. Strip trailing whitespace - Removes any space characters after the end of the last non-space character + Remove any space characters after the last non-space character of a line. .. index:: - single: Import module single: Run script - Run menu (Editor window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python Shell - Open or wake up the Python Shell window + Open or wake up the Python Shell window. -Check module +Check Module Check the syntax of the module currently open in the Editor window. If the - module has not been saved IDLE will prompt the user to save the code. + module has not been saved IDLE will either prompt the user to save or + autosave, as selected in the General tab of the Idle Settings dialog. If + there is a syntax error, the approximate location is indicated in the + Editor window. -Run module - Restart the shell to clean the environment, then execute the currently - open module. If the module has not been saved IDLE will prompt the user - to save the code. +Run Module + Do Check Module (above). If no error, restart the shell to clean the + environment, then execute the module. Shell menu (Shell window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ View Last Restart - Scroll the shell window to the last Shell restart + Scroll the shell window to the last Shell restart. Restart Shell - Restart the shell to clean the environment + Restart the shell to clean the environment. Debug menu (Shell window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Go to file/line - Look around the insert point for a filename and line number, open the file, - and show the line. Useful to view the source lines referenced in an - exception traceback. Available in the context menu of the Shell window. +Go to File/Line + Look on the current line. with the cursor, and the line above for a filename + and line number. If found, open the file if not already open, and show the + line. Use this to view source lines referenced in an exception traceback + and lines found by Find in Files. Also available in the context menu of + the Shell window and Output windows. + +.. index:: + single: debugger + single: stack viewer Debugger (toggle) - This feature is not complete and considered experimental. Run commands in - the shell under the debugger + When actived, code entered in the Shell or run from an Editor will run + under the debugger. In the Editor, breakpoints can be set with the context + menu. This feature is still incomplete and somewhat experimental. -Stack viewer - Show the stack traceback of the last exception +Stack Viewer + Show the stack traceback of the last exception in a tree widget, with + access to locals and globals. Auto-open Stack Viewer - Toggle automatically opening the stack viewer on unhandled exception - -.. index:: - single: stack viewer - single: debugger + Toggle automatically opening the stack viewer on an unhandled exception. Options menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -229,23 +245,26 @@ Configure IDLE Open a configuration dialog. Fonts, indentation, keybindings, and color themes may be altered. Startup Preferences may be set, and additional - help sources can be specified. + help sources can be specified. Non-default user setting are saved in a + .idlerc directory in the user's home directory. Problems caused by bad user + configuration files are solved by editing or deleting one or more of the + files in .idlerc. Configure Extensions Open a configuration dialog for setting preferences for extensions - (discussed below). + (discussed below). See note above about the location of user settings. Code Context (toggle)(Editor Window only) Open a pane at the top of the edit window which shows the block context - of the section of code which is scrolling off the top of the window. + of the code which has scrolled above the top of the window. Windows menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Zoom Height - Toggles the window between normal size (40x80 initial setting) and maximum - height. The initial size is in the Configure IDLE dialog under the general - tab. + Toggles the window between normal size and maximum height. The initial size + defaults to 40 lines by 80 chars unless changed on the General tab of the + Configure IDLE dialog. The rest of this menu lists the names of all open windows; select one to bring it to the foreground (deiconifying it if necessary). @@ -254,16 +273,19 @@ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ About IDLE - Version, copyright, license, credits + Display version, copyright, license, credits, and more. IDLE Help Display a help file for IDLE detailing the menu options, basic editing and navigation, and other tips. Python Docs - Access local Python documentation, if installed. Or will start a web browser + Access local Python documentation, if installed, or start a web browser and open docs.python.org showing the latest Python documentation. +Turtle Demo + Run the turtledemo module with example python code and turtle drawings. + Additional help sources may be added here with the Configure IDLE dialog under the General tab. @@ -275,39 +297,32 @@ single: Clear Breakpoint single: breakpoints -Editor Window context menu +Context Menus ^^^^^^^^^^^^^^^^^^^^^^^^^^ -* Right-click in Editor window (Control-click on OS X) +Open a context menu by right-clicking in a window (Control-click on OS X). +Context menus have the standard clipboard functions also on the Edit menu. Cut - Copy selection into system-wide clipboard; then delete selection + Copy selection into the system-wide clipboard; then delete the selection. Copy - Copy selection into system-wide clipboard + Copy selection into the system-wide clipboard. Paste - Insert system-wide clipboard into window + Insert contents of the system-wide clipboard into the current window. + +Editor windows also have breakpoint functions. Lines with a breakpoint set are +specially marked. Breakpoints only have an effect when running under the +debugger. Breakpoints for a file are saved in the user's .idlerc directory. Set Breakpoint - Sets a breakpoint. Breakpoints are only enabled when the debugger is open. + Set a breakpoint on the current line. Clear Breakpoint - Clears the breakpoint on that line. + Clear the breakpoint on that line. -Shell Window context menu -^^^^^^^^^^^^^^^^^^^^^^^^^ - -* Right-click in Python Shell window (Control-click on OS X) - -Cut - Copy selection into system-wide clipboard; then delete selection - -Copy - Copy selection into system-wide clipboard - -Paste - Insert system-wide clipboard into window +Shell and Output windows have the following. Go to file/line Same as in Debug menu. @@ -481,8 +496,8 @@ In addition, ``Tk`` also loads a startup file if it is present. Note that the Tk file is loaded unconditionally. This additional file is ``.Idle.py`` and is looked for in the user's home directory. Statements in this file will be -executed in the Tk namespace, so this file is not useful for importing functions -to be used from IDLE's Python shell. +executed in the Tk namespace, so this file is not useful for importing +functions to be used from IDLE's Python shell. Command line usage @@ -508,9 +523,9 @@ #. Otherwise, if neither ``-e`` nor ``-c`` is used, the first argument is a script which is executed with the remaining arguments in - ``sys.argv[1:...]`` and ``sys.argv[0]`` set to the script name. If the script - name is '-', no script is executed but an interactive Python session is started; - the arguments are still available in ``sys.argv``. + ``sys.argv[1:...]`` and ``sys.argv[0]`` set to the script name. If the + script name is '-', no script is executed but an interactive Python session + is started; the arguments are still available in ``sys.argv``. Running without a subprocess ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 08:43:49 2014 From: python-checkins at python.org (terry.reedy) Date: Fri, 05 Dec 2014 07:43:49 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4=2E_Edit_Menus_section_of_Idle_doc=2E?= Message-ID: <20141205074349.27491.4050@psf.io> https://hg.python.org/cpython/rev/b2889ab48d11 changeset: 93734:b2889ab48d11 parent: 93731:e968803baaa4 parent: 93733:c3c060e5ba99 user: Terry Jan Reedy date: Fri Dec 05 02:43:29 2014 -0500 summary: Merge with 3.4. Edit Menus section of Idle doc. files: Doc/library/idle.rst | 261 ++++++++++++++++-------------- 1 files changed, 138 insertions(+), 123 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -29,136 +29,147 @@ Menus ----- -IDLE has two window types, the Shell window and the Editor window. It is -possible to have multiple editor windows simultaneously. IDLE's -menus dynamically change based on which window is currently selected. Each menu -documented below indicates which window type it is associated with. Click on -the dotted line at the top of a menu to "tear it off": a separate window -containing the menu is created (for Unix and Windows only). +IDLE has two main window types, the Shell window and the Editor window. It is +possible to have multiple editor windows simultaneously. Output windows, such +as used for Edit / Find in Files, are a subtype of edit window. They currently +have the same top menu as Editor windows but a different default title and +context menu. + +IDLE's menus dynamically change based on which window is currently selected. +Each menu documented below indicates which window type it is associated with. +Click on the dotted line at the top of a menu to "tear it off": a separate +window containing the menu is created (for Unix and Windows only). File menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -New file - Create a new file editing window +New File + Create a new file editing window. Open... - Open an existing file - -Open module... - Open an existing module (searches sys.path) + Open an existing file with an Open dialog. Recent Files - Open a list of recent files + Open a list of recent files. Click one to open it. + +Open Module... + Open an existing module (searches sys.path). .. index:: single: Class browser single: Path browser -Class browser - Show classes and methods in current file +Class Browser + Show functions, classes, and methods in the current Editor file in a + tree structure. In the shell, open a module first. -Path browser - Show sys.path directories, modules, classes and methods +Path Browser + Show sys.path directories, modules, functions, classes and methods in a + tree structure. Save - Save current window to the associated file (unsaved windows have a - \* before and after the window title) + Save the current window to the associated file, if there is one. Windows + that have been changed since being opened or last saved have a \* before + and after the window title. If there is no associated file, + do Save As instead. Save As... - Save current window to new file, which becomes the associated file + Save the current window with a Save As dialog. The file saved becomes the + new associated file for the window. Save Copy As... - Save current window to different file without changing the associated file + Save the current window to different file without changing the associated + file. Print Window - Print the current window + Print the current window to the default printer. Close - Close current window (asks to save if unsaved) + Close the current window (ask to save if unsaved). Exit - Close all windows and quit IDLE (asks to save if unsaved) - + Close all windows and quit IDLE (ask to save unsaved windows). Edit menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Undo - Undo last change to current window (a maximum of 1000 changes may be undone) + Undo the last change to the current window. A maximum of 1000 changes may + be undone. Redo - Redo last undone change to current window + Redo the last undone change to the current window. Cut - Copy selection into system-wide clipboard; then delete the selection + Copy selection into the system-wide clipboard; then delete the selection. Copy - Copy selection into system-wide clipboard + Copy selection into the system-wide clipboard. Paste - Insert system-wide clipboard into window + Insert contents of the system-wide clipboard into the current window. + +The clipboard functions are also available in context menus. Select All - Select the entire contents of the edit buffer + Select the entire contents of the current window. Find... - Open a search dialog box with many options + Open a search dialog with many options -Find again - Repeat last search +Find Again + Repeat the last search, if there is one. -Find selection - Search for the string in the selection +Find Selection + Search for the currently selected string, if there is one. Find in Files... - Open a search dialog box for searching files + Open a file search dialog. Put results in an new output window. Replace... - Open a search-and-replace dialog box + Open a search-and-replace dialog. -Go to line - Ask for a line number and show that line +Go to Line + Move cursor to the line number requested and make that line visible. -Expand word - Expand the word you have typed to match another word in the same buffer; - repeat to get a different expansion +Show Completions + Open a scrollable list allowing selection of keywords and attributes. See + Completions in the Tips sections below. + +Expand Word + Expand a prefix you have typed to match a full word in the same window; + repeat to get a different expansion. Show call tip After an unclosed parenthesis for a function, open a small window with - function parameter hints + function parameter hints. Show surrounding parens - Highlight the surrounding parenthesis - -Show Completions - Open a scroll window allowing selection keywords and attributes. See - Completions below. - + Highlight the surrounding parenthesis. Format menu (Editor window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Indent region - Shift selected lines right by the indent width (default 4 spaces) +Indent Region + Shift selected lines right by the indent width (default 4 spaces). -Dedent region - Shift selected lines left by the indent width (default 4 spaces) +Dedent Region + Shift selected lines left by the indent width (default 4 spaces). -Comment out region - Insert ## in front of selected lines +Comment Out Region + Insert ## in front of selected lines. -Uncomment region - Remove leading # or ## from selected lines +Uncomment Region + Remove leading # or ## from selected lines. -Tabify region - Turns *leading* stretches of spaces into tabs. (Note: We recommend using +Tabify Region + Turn *leading* stretches of spaces into tabs. (Note: We recommend using 4 space blocks to indent Python code.) -Untabify region - Turn *all* tabs into the correct number of spaces +Untabify Region + Turn *all* tabs into the correct number of spaces. -Toggle tabs +Toggle Tabs Open a dialog to switch between indenting with spaces and tabs. New Indent Width @@ -166,62 +177,67 @@ community is 4 spaces. Format Paragraph - Reformat the current blank-line-separated paragraph. All lines in the - paragraph will be formatted to less than 80 columns. + Reformat the current blank-line-delimited paragraph in comment block or + multiline string or selected line in a string. All lines in the + paragraph will be formatted to less than N columns, where N defaults to 72. Strip trailing whitespace - Removes any space characters after the end of the last non-space character + Remove any space characters after the last non-space character of a line. .. index:: - single: Import module single: Run script - Run menu (Editor window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python Shell - Open or wake up the Python Shell window + Open or wake up the Python Shell window. -Check module +Check Module Check the syntax of the module currently open in the Editor window. If the - module has not been saved IDLE will prompt the user to save the code. + module has not been saved IDLE will either prompt the user to save or + autosave, as selected in the General tab of the Idle Settings dialog. If + there is a syntax error, the approximate location is indicated in the + Editor window. -Run module - Restart the shell to clean the environment, then execute the currently - open module. If the module has not been saved IDLE will prompt the user - to save the code. +Run Module + Do Check Module (above). If no error, restart the shell to clean the + environment, then execute the module. Shell menu (Shell window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ View Last Restart - Scroll the shell window to the last Shell restart + Scroll the shell window to the last Shell restart. Restart Shell - Restart the shell to clean the environment + Restart the shell to clean the environment. Debug menu (Shell window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Go to file/line - Look around the insert point for a filename and line number, open the file, - and show the line. Useful to view the source lines referenced in an - exception traceback. Available in the context menu of the Shell window. +Go to File/Line + Look on the current line. with the cursor, and the line above for a filename + and line number. If found, open the file if not already open, and show the + line. Use this to view source lines referenced in an exception traceback + and lines found by Find in Files. Also available in the context menu of + the Shell window and Output windows. + +.. index:: + single: debugger + single: stack viewer Debugger (toggle) - This feature is not complete and considered experimental. Run commands in - the shell under the debugger + When actived, code entered in the Shell or run from an Editor will run + under the debugger. In the Editor, breakpoints can be set with the context + menu. This feature is still incomplete and somewhat experimental. -Stack viewer - Show the stack traceback of the last exception +Stack Viewer + Show the stack traceback of the last exception in a tree widget, with + access to locals and globals. Auto-open Stack Viewer - Toggle automatically opening the stack viewer on unhandled exception - -.. index:: - single: stack viewer - single: debugger + Toggle automatically opening the stack viewer on an unhandled exception. Options menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -229,23 +245,26 @@ Configure IDLE Open a configuration dialog. Fonts, indentation, keybindings, and color themes may be altered. Startup Preferences may be set, and additional - help sources can be specified. + help sources can be specified. Non-default user setting are saved in a + .idlerc directory in the user's home directory. Problems caused by bad user + configuration files are solved by editing or deleting one or more of the + files in .idlerc. Configure Extensions Open a configuration dialog for setting preferences for extensions - (discussed below). + (discussed below). See note above about the location of user settings. Code Context (toggle)(Editor Window only) Open a pane at the top of the edit window which shows the block context - of the section of code which is scrolling off the top of the window. + of the code which has scrolled above the top of the window. Windows menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Zoom Height - Toggles the window between normal size (40x80 initial setting) and maximum - height. The initial size is in the Configure IDLE dialog under the general - tab. + Toggles the window between normal size and maximum height. The initial size + defaults to 40 lines by 80 chars unless changed on the General tab of the + Configure IDLE dialog. The rest of this menu lists the names of all open windows; select one to bring it to the foreground (deiconifying it if necessary). @@ -254,16 +273,19 @@ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ About IDLE - Version, copyright, license, credits + Display version, copyright, license, credits, and more. IDLE Help Display a help file for IDLE detailing the menu options, basic editing and navigation, and other tips. Python Docs - Access local Python documentation, if installed. Or will start a web browser + Access local Python documentation, if installed, or start a web browser and open docs.python.org showing the latest Python documentation. +Turtle Demo + Run the turtledemo module with example python code and turtle drawings. + Additional help sources may be added here with the Configure IDLE dialog under the General tab. @@ -275,39 +297,32 @@ single: Clear Breakpoint single: breakpoints -Editor Window context menu +Context Menus ^^^^^^^^^^^^^^^^^^^^^^^^^^ -* Right-click in Editor window (Control-click on OS X) +Open a context menu by right-clicking in a window (Control-click on OS X). +Context menus have the standard clipboard functions also on the Edit menu. Cut - Copy selection into system-wide clipboard; then delete selection + Copy selection into the system-wide clipboard; then delete the selection. Copy - Copy selection into system-wide clipboard + Copy selection into the system-wide clipboard. Paste - Insert system-wide clipboard into window + Insert contents of the system-wide clipboard into the current window. + +Editor windows also have breakpoint functions. Lines with a breakpoint set are +specially marked. Breakpoints only have an effect when running under the +debugger. Breakpoints for a file are saved in the user's .idlerc directory. Set Breakpoint - Sets a breakpoint. Breakpoints are only enabled when the debugger is open. + Set a breakpoint on the current line. Clear Breakpoint - Clears the breakpoint on that line. + Clear the breakpoint on that line. -Shell Window context menu -^^^^^^^^^^^^^^^^^^^^^^^^^ - -* Right-click in Python Shell window (Control-click on OS X) - -Cut - Copy selection into system-wide clipboard; then delete selection - -Copy - Copy selection into system-wide clipboard - -Paste - Insert system-wide clipboard into window +Shell and Output windows have the following. Go to file/line Same as in Debug menu. @@ -481,8 +496,8 @@ In addition, ``Tk`` also loads a startup file if it is present. Note that the Tk file is loaded unconditionally. This additional file is ``.Idle.py`` and is looked for in the user's home directory. Statements in this file will be -executed in the Tk namespace, so this file is not useful for importing functions -to be used from IDLE's Python shell. +executed in the Tk namespace, so this file is not useful for importing +functions to be used from IDLE's Python shell. Command line usage @@ -508,9 +523,9 @@ #. Otherwise, if neither ``-e`` nor ``-c`` is used, the first argument is a script which is executed with the remaining arguments in - ``sys.argv[1:...]`` and ``sys.argv[0]`` set to the script name. If the script - name is '-', no script is executed but an interactive Python session is started; - the arguments are still available in ``sys.argv``. + ``sys.argv[1:...]`` and ``sys.argv[0]`` set to the script name. If the + script name is '-', no script is executed but an interactive Python session + is started; the arguments are still available in ``sys.argv``. Running without a subprocess ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 08:43:49 2014 From: python-checkins at python.org (terry.reedy) Date: Fri, 05 Dec 2014 07:43:49 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Edit_Menus_sec?= =?utf-8?q?tion_of_Idle_doc=2E__Make_menu_entries_match_current_Idle_in_or?= =?utf-8?q?der?= Message-ID: <20141205074349.74611.88017@psf.io> https://hg.python.org/cpython/rev/84928af5d703 changeset: 93732:84928af5d703 branch: 2.7 parent: 93718:395673aac686 user: Terry Jan Reedy date: Fri Dec 05 02:42:54 2014 -0500 summary: Edit Menus section of Idle doc. Make menu entries match current Idle in order and case. Edit some of the explanatory sentences and end all with a period. files: Doc/library/idle.rst | 261 ++++++++++++++++-------------- 1 files changed, 138 insertions(+), 123 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -29,136 +29,147 @@ Menus ----- -IDLE has two window types, the Shell window and the Editor window. It is -possible to have multiple editor windows simultaneously. IDLE's -menus dynamically change based on which window is currently selected. Each menu -documented below indicates which window type it is associated with. Click on -the dotted line at the top of a menu to "tear it off": a separate window -containing the menu is created (for Unix and Windows only). +IDLE has two main window types, the Shell window and the Editor window. It is +possible to have multiple editor windows simultaneously. Output windows, such +as used for Edit / Find in Files, are a subtype of edit window. They currently +have the same top menu as Editor windows but a different default title and +context menu. + +IDLE's menus dynamically change based on which window is currently selected. +Each menu documented below indicates which window type it is associated with. +Click on the dotted line at the top of a menu to "tear it off": a separate +window containing the menu is created (for Unix and Windows only). File menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -New file - Create a new file editing window +New File + Create a new file editing window. Open... - Open an existing file - -Open module... - Open an existing module (searches sys.path) + Open an existing file with an Open dialog. Recent Files - Open a list of recent files + Open a list of recent files. Click one to open it. + +Open Module... + Open an existing module (searches sys.path). .. index:: single: Class browser single: Path browser -Class browser - Show classes and methods in current file +Class Browser + Show functions, classes, and methods in the current Editor file in a + tree structure. In the shell, open a module first. -Path browser - Show sys.path directories, modules, classes and methods +Path Browser + Show sys.path directories, modules, functions, classes and methods in a + tree structure. Save - Save current window to the associated file (unsaved windows have a - \* before and after the window title) + Save the current window to the associated file, if there is one. Windows + that have been changed since being opened or last saved have a \* before + and after the window title. If there is no associated file, + do Save As instead. Save As... - Save current window to new file, which becomes the associated file + Save the current window with a Save As dialog. The file saved becomes the + new associated file for the window. Save Copy As... - Save current window to different file without changing the associated file + Save the current window to different file without changing the associated + file. Print Window - Print the current window + Print the current window to the default printer. Close - Close current window (asks to save if unsaved) + Close the current window (ask to save if unsaved). Exit - Close all windows and quit IDLE (asks to save if unsaved) - + Close all windows and quit IDLE (ask to save unsaved windows). Edit menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Undo - Undo last change to current window (a maximum of 1000 changes may be undone) + Undo the last change to the current window. A maximum of 1000 changes may + be undone. Redo - Redo last undone change to current window + Redo the last undone change to the current window. Cut - Copy selection into system-wide clipboard; then delete the selection + Copy selection into the system-wide clipboard; then delete the selection. Copy - Copy selection into system-wide clipboard + Copy selection into the system-wide clipboard. Paste - Insert system-wide clipboard into window + Insert contents of the system-wide clipboard into the current window. + +The clipboard functions are also available in context menus. Select All - Select the entire contents of the edit buffer + Select the entire contents of the current window. Find... - Open a search dialog box with many options + Open a search dialog with many options -Find again - Repeat last search +Find Again + Repeat the last search, if there is one. -Find selection - Search for the string in the selection +Find Selection + Search for the currently selected string, if there is one. Find in Files... - Open a search dialog box for searching files + Open a file search dialog. Put results in an new output window. Replace... - Open a search-and-replace dialog box + Open a search-and-replace dialog. -Go to line - Ask for a line number and show that line +Go to Line + Move cursor to the line number requested and make that line visible. -Expand word - Expand the word you have typed to match another word in the same buffer; - repeat to get a different expansion +Show Completions + Open a scrollable list allowing selection of keywords and attributes. See + Completions in the Tips sections below. + +Expand Word + Expand a prefix you have typed to match a full word in the same window; + repeat to get a different expansion. Show call tip After an unclosed parenthesis for a function, open a small window with - function parameter hints + function parameter hints. Show surrounding parens - Highlight the surrounding parenthesis - -Show Completions - Open a scroll window allowing selection keywords and attributes. See - Completions below. - + Highlight the surrounding parenthesis. Format menu (Editor window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Indent region - Shift selected lines right by the indent width (default 4 spaces) +Indent Region + Shift selected lines right by the indent width (default 4 spaces). -Dedent region - Shift selected lines left by the indent width (default 4 spaces) +Dedent Region + Shift selected lines left by the indent width (default 4 spaces). -Comment out region - Insert ## in front of selected lines +Comment Out Region + Insert ## in front of selected lines. -Uncomment region - Remove leading # or ## from selected lines +Uncomment Region + Remove leading # or ## from selected lines. -Tabify region - Turns *leading* stretches of spaces into tabs. (Note: We recommend using +Tabify Region + Turn *leading* stretches of spaces into tabs. (Note: We recommend using 4 space blocks to indent Python code.) -Untabify region - Turn *all* tabs into the correct number of spaces +Untabify Region + Turn *all* tabs into the correct number of spaces. -Toggle tabs +Toggle Tabs Open a dialog to switch between indenting with spaces and tabs. New Indent Width @@ -166,62 +177,67 @@ community is 4 spaces. Format Paragraph - Reformat the current blank-line-separated paragraph. All lines in the - paragraph will be formatted to less than 80 columns. + Reformat the current blank-line-delimited paragraph in comment block or + multiline string or selected line in a string. All lines in the + paragraph will be formatted to less than N columns, where N defaults to 72. Strip trailing whitespace - Removes any space characters after the end of the last non-space character + Remove any space characters after the last non-space character of a line. .. index:: - single: Import module single: Run script - Run menu (Editor window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python Shell - Open or wake up the Python Shell window + Open or wake up the Python Shell window. -Check module +Check Module Check the syntax of the module currently open in the Editor window. If the - module has not been saved IDLE will prompt the user to save the code. + module has not been saved IDLE will either prompt the user to save or + autosave, as selected in the General tab of the Idle Settings dialog. If + there is a syntax error, the approximate location is indicated in the + Editor window. -Run module - Restart the shell to clean the environment, then execute the currently - open module. If the module has not been saved IDLE will prompt the user - to save the code. +Run Module + Do Check Module (above). If no error, restart the shell to clean the + environment, then execute the module. Shell menu (Shell window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ View Last Restart - Scroll the shell window to the last Shell restart + Scroll the shell window to the last Shell restart. Restart Shell - Restart the shell to clean the environment + Restart the shell to clean the environment. Debug menu (Shell window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Go to file/line - Look around the insert point for a filename and line number, open the file, - and show the line. Useful to view the source lines referenced in an - exception traceback. Available in the context menu of the Shell window. +Go to File/Line + Look on the current line. with the cursor, and the line above for a filename + and line number. If found, open the file if not already open, and show the + line. Use this to view source lines referenced in an exception traceback + and lines found by Find in Files. Also available in the context menu of + the Shell window and Output windows. + +.. index:: + single: debugger + single: stack viewer Debugger (toggle) - This feature is not complete and considered experimental. Run commands in - the shell under the debugger + When actived, code entered in the Shell or run from an Editor will run + under the debugger. In the Editor, breakpoints can be set with the context + menu. This feature is still incomplete and somewhat experimental. -Stack viewer - Show the stack traceback of the last exception +Stack Viewer + Show the stack traceback of the last exception in a tree widget, with + access to locals and globals. Auto-open Stack Viewer - Toggle automatically opening the stack viewer on unhandled exception - -.. index:: - single: stack viewer - single: debugger + Toggle automatically opening the stack viewer on an unhandled exception. Options menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -229,23 +245,26 @@ Configure IDLE Open a configuration dialog. Fonts, indentation, keybindings, and color themes may be altered. Startup Preferences may be set, and additional - help sources can be specified. + help sources can be specified. Non-default user setting are saved in a + .idlerc directory in the user's home directory. Problems caused by bad user + configuration files are solved by editing or deleting one or more of the + files in .idlerc. Configure Extensions Open a configuration dialog for setting preferences for extensions - (discussed below). + (discussed below). See note above about the location of user settings. Code Context (toggle)(Editor Window only) Open a pane at the top of the edit window which shows the block context - of the section of code which is scrolling off the top of the window. + of the code which has scrolled above the top of the window. Windows menu (Shell and Editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Zoom Height - Toggles the window between normal size (40x80 initial setting) and maximum - height. The initial size is in the Configure IDLE dialog under the general - tab. + Toggles the window between normal size and maximum height. The initial size + defaults to 40 lines by 80 chars unless changed on the General tab of the + Configure IDLE dialog. The rest of this menu lists the names of all open windows; select one to bring it to the foreground (deiconifying it if necessary). @@ -254,16 +273,19 @@ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ About IDLE - Version, copyright, license, credits + Display version, copyright, license, credits, and more. IDLE Help Display a help file for IDLE detailing the menu options, basic editing and navigation, and other tips. Python Docs - Access local Python documentation, if installed. Or will start a web browser + Access local Python documentation, if installed, or start a web browser and open docs.python.org showing the latest Python documentation. +Turtle Demo + Run the turtledemo module with example python code and turtle drawings. + Additional help sources may be added here with the Configure IDLE dialog under the General tab. @@ -275,39 +297,32 @@ single: Clear Breakpoint single: breakpoints -Editor Window context menu +Context Menus ^^^^^^^^^^^^^^^^^^^^^^^^^^ -* Right-click in Editor window (Control-click on OS X) +Open a context menu by right-clicking in a window (Control-click on OS X). +Context menus have the standard clipboard functions also on the Edit menu. Cut - Copy selection into system-wide clipboard; then delete selection + Copy selection into the system-wide clipboard; then delete the selection. Copy - Copy selection into system-wide clipboard + Copy selection into the system-wide clipboard. Paste - Insert system-wide clipboard into window + Insert contents of the system-wide clipboard into the current window. + +Editor windows also have breakpoint functions. Lines with a breakpoint set are +specially marked. Breakpoints only have an effect when running under the +debugger. Breakpoints for a file are saved in the user's .idlerc directory. Set Breakpoint - Sets a breakpoint. Breakpoints are only enabled when the debugger is open. + Set a breakpoint on the current line. Clear Breakpoint - Clears the breakpoint on that line. + Clear the breakpoint on that line. -Shell Window context menu -^^^^^^^^^^^^^^^^^^^^^^^^^ - -* Right-click in Python Shell window (Control-click on OS X) - -Cut - Copy selection into system-wide clipboard; then delete selection - -Copy - Copy selection into system-wide clipboard - -Paste - Insert system-wide clipboard into window +Shell and Output windows have the following. Go to file/line Same as in Debug menu. @@ -481,8 +496,8 @@ In addition, ``Tk`` also loads a startup file if it is present. Note that the Tk file is loaded unconditionally. This additional file is ``.Idle.py`` and is looked for in the user's home directory. Statements in this file will be -executed in the Tk namespace, so this file is not useful for importing functions -to be used from IDLE's Python shell. +executed in the Tk namespace, so this file is not useful for importing +functions to be used from IDLE's Python shell. Command line usage @@ -508,9 +523,9 @@ #. Otherwise, if neither ``-e`` nor ``-c`` is used, the first argument is a script which is executed with the remaining arguments in - ``sys.argv[1:...]`` and ``sys.argv[0]`` set to the script name. If the script - name is '-', no script is executed but an interactive Python session is started; - the arguments are still available in ``sys.argv``. + ``sys.argv[1:...]`` and ``sys.argv[0]`` set to the script name. If the + script name is '-', no script is executed but an interactive Python session + is started; the arguments are still available in ``sys.argv``. Running without a subprocess ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- Repository URL: https://hg.python.org/cpython From yselivanov.ml at gmail.com Fri Dec 5 06:43:19 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 05 Dec 2014 00:43:19 -0500 Subject: [Python-checkins] cpython: docs.inspect: Fix BoundArguments example. Issue #22998. In-Reply-To: <548135CA.9070702@udel.edu> References: <20141205034901.74601.18400@psf.io> <548135CA.9070702@udel.edu> Message-ID: <548145F7.1060804@gmail.com> Thank you, Terry. Yury On 2014-12-04, 11:34 PM, Terry Reedy wrote: > On 12/4/2014 10:49 PM, yury.selivanov wrote: >> https://hg.python.org/cpython/rev/697adefaba6b >> changeset: 93730:697adefaba6b >> parent: 93728:2fa3c68de07e >> user: Yury Selivanov >> date: Thu Dec 04 22:48:47 2014 -0500 >> summary: >> docs.inspect: Fix BoundArguments example. Issue #22998. >> >> files: >> Doc/library/inspect.rst | 3 ++- >> 1 files changed, 2 insertions(+), 1 deletions(-) >> >> >> diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst >> --- a/Doc/library/inspect.rst >> +++ b/Doc/library/inspect.rst >> @@ -678,7 +678,8 @@ >> ((5,), {}) >> >> >>> for param in sig.parameters.values(): >> - ... if param.name not in ba.arguments: >> + ... if (param.name not in ba.arguments >> + ... and param.default is not param.empty): >> ... ba.arguments[param.name] = param.default >> >> >>> ba.args, ba.kwargs > > Yury, it appears that you patched 3.5 directly instead of merging the > 3.4 commit 71c38c233e5c into 3.5. The result was that the 3.4 commit > was left unmerged, potentially interfering with the next person doing > a merge. I believe I fixed it with what seems effectively to be a > null merge. > > Terry > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins From python-checkins at python.org Fri Dec 5 10:25:47 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 05 Dec 2014 09:25:47 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNTk5?= =?utf-8?q?=3A_Enhance_tokenize=2Eopen=28=29_to_be_able_to_call_it_during_?= =?utf-8?q?Python?= Message-ID: <20141205092547.73380.66595@psf.io> https://hg.python.org/cpython/rev/cd3244416592 changeset: 93735:cd3244416592 branch: 3.4 parent: 93733:c3c060e5ba99 user: Victor Stinner date: Fri Dec 05 10:17:10 2014 +0100 summary: Issue #22599: Enhance tokenize.open() to be able to call it during Python finalization. Before the module kept a reference to the builtins module, but the module attributes are cleared during Python finalization. Instead, keep directly a reference to the open() function. This enhancement is not perfect, calling tokenize.open() can still fail if called very late during Python finalization. Usually, the function is called by the linecache module which is called to display a traceback or emit a warning. files: Lib/tokenize.py | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/tokenize.py b/Lib/tokenize.py --- a/Lib/tokenize.py +++ b/Lib/tokenize.py @@ -24,7 +24,6 @@ __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, ' 'Skip Montanaro, Raymond Hettinger, Trent Nelson, ' 'Michael Foord') -import builtins from codecs import lookup, BOM_UTF8 import collections from io import TextIOWrapper @@ -430,11 +429,13 @@ return default, [first, second] +_builtin_open = open + def open(filename): """Open a file in read only mode using the encoding detected by detect_encoding(). """ - buffer = builtins.open(filename, 'rb') + buffer = _builtin_open(filename, 'rb') encoding, lines = detect_encoding(buffer.readline) buffer.seek(0) text = TextIOWrapper(buffer, encoding, line_buffering=True) @@ -657,7 +658,7 @@ # Tokenize the input if args.filename: filename = args.filename - with builtins.open(filename, 'rb') as f: + with _builtin_open(filename, 'rb') as f: tokens = list(tokenize(f.readline)) else: filename = "" -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 10:25:47 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 05 Dec 2014 09:25:47 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2322599=3A_Enhance_tokenize=2Eo?= =?utf-8?q?pen=28=29_to_be_able_to_call_it_during?= Message-ID: <20141205092547.74585.83538@psf.io> https://hg.python.org/cpython/rev/22f2784a276e changeset: 93736:22f2784a276e parent: 93734:b2889ab48d11 parent: 93735:cd3244416592 user: Victor Stinner date: Fri Dec 05 10:18:30 2014 +0100 summary: (Merge 3.4) Issue #22599: Enhance tokenize.open() to be able to call it during Python finalization. Before the module kept a reference to the builtins module, but the module attributes are cleared during Python finalization. Instead, keep directly a reference to the open() function. This enhancement is not perfect, calling tokenize.open() can still fail if called very late during Python finalization. Usually, the function is called by the linecache module which is called to display a traceback or emit a warning. files: Lib/test/test_traceback.py | 33 ++++++++++++++++++++++++++ Lib/tokenize.py | 7 +++-- 2 files changed, 37 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -6,6 +6,8 @@ import re from test.support import run_unittest, Error, captured_output from test.support import TESTFN, unlink, cpython_only +from test.script_helper import assert_python_ok +import textwrap import traceback @@ -169,6 +171,37 @@ # Issue #18960: coding spec should has no effect do_test("0\n# coding: GBK\n", "h\xe9 ho", 'utf-8', 5) + def test_print_traceback_at_exit(self): + # Issue #22599: Ensure that it is possible to use the traceback module + # to display an exception at Python exit + code = textwrap.dedent(""" + import sys + import traceback + + class PrintExceptionAtExit(object): + def __init__(self): + try: + x = 1 / 0 + except Exception: + self.exc_info = sys.exc_info() + # self.exc_info[1] (traceback) contains frames: + # explicitly clear the reference to self in the current + # frame to break a reference cycle + self = None + + def __del__(self): + traceback.print_exception(*self.exc_info) + + # Keep a reference in the module namespace to call the destructor + # when the module is unloaded + obj = PrintExceptionAtExit() + """) + rc, stdout, stderr = assert_python_ok('-c', code) + expected = [b'Traceback (most recent call last):', + b' File "", line 8, in __init__', + b'ZeroDivisionError: division by zero'] + self.assertEqual(stderr.splitlines(), expected) + class TracebackFormatTests(unittest.TestCase): diff --git a/Lib/tokenize.py b/Lib/tokenize.py --- a/Lib/tokenize.py +++ b/Lib/tokenize.py @@ -24,7 +24,6 @@ __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, ' 'Skip Montanaro, Raymond Hettinger, Trent Nelson, ' 'Michael Foord') -import builtins from codecs import lookup, BOM_UTF8 import collections from io import TextIOWrapper @@ -431,11 +430,13 @@ return default, [first, second] +_builtin_open = open + def open(filename): """Open a file in read only mode using the encoding detected by detect_encoding(). """ - buffer = builtins.open(filename, 'rb') + buffer = _builtin_open(filename, 'rb') encoding, lines = detect_encoding(buffer.readline) buffer.seek(0) text = TextIOWrapper(buffer, encoding, line_buffering=True) @@ -658,7 +659,7 @@ # Tokenize the input if args.filename: filename = args.filename - with builtins.open(filename, 'rb') as f: + with _builtin_open(filename, 'rb') as f: tokens = list(tokenize(f.readline)) else: filename = "" -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Fri Dec 5 10:31:23 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 05 Dec 2014 10:31:23 +0100 Subject: [Python-checkins] Daily reference leaks (2fa3c68de07e): sum=3 Message-ID: results for 2fa3c68de07e on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogJMlIaH', '-x'] From python-checkins at python.org Fri Dec 5 17:01:59 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 05 Dec 2014 16:01:59 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <20141205160144.73378.72700@psf.io> https://hg.python.org/cpython/rev/021c1df36910 changeset: 93738:021c1df36910 parent: 93736:22f2784a276e parent: 93737:55b94462ca7f user: Brett Cannon date: Fri Dec 05 11:01:30 2014 -0500 summary: Merge with 3.4 files: Doc/howto/pyporting.rst | 803 +++++++++------------------ 1 files changed, 281 insertions(+), 522 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -19,600 +19,359 @@ If you would like to read one core Python developer's take on why Python 3 came into existence, you can read Nick Coghlan's `Python 3 Q & A`_. - If you prefer to read a (free) book on porting a project to Python 3, - consider reading `Porting to Python 3`_ by Lennart Regebro which should cover - much of what is discussed in this HOWTO. - For help with porting, you can email the python-porting_ mailing list with questions. -The Short Version -================= +The Short Explanation +===================== -* Decide what's the oldest version of Python 2 you want to support (if at all) -* Make sure you have a thorough test suite and use continuous integration - testing to make sure you stay compatible with the versions of Python you care - about -* If you have dependencies, check their Python 3 status using caniusepython3 - (`command-line tool `__, - `web app `__) +To make your project be single-source Python 2/3 compatible, the basic steps +are: -With that done, your options are: +#. Update your code to drop support for Python 2.5 or older (supporting only + Python 2.7 is ideal) +#. Make sure you have good test coverage (coverage.py_ can help) +#. Learn the differences between Python 2 & 3 +#. Use Modernize_ or Futurize_ to update your code +#. Use Pylint_ to help make sure you don't regress on your Python 3 support + (if only supporting Python 2.7/3.4 or newer) +#. Use caniusepython3_ to find out which of your dependencies are blocking your + use of Python 3 +#. Once your dependencies are no longer blocking you, use continuous integration + to make sure you stay compatible with Python 2 & 3 (tox_ can help test + against multiple versions of Python) -* If you are dropping Python 2 support, use :ref:`2to3 <2to3-reference>` to port - to Python 3 +If you are dropping support for Python 2 entirely, then after you learn the +differences between Python 2 & 3 you can run 2to3_ over your code and skip the +rest of the steps outlined above. -* If you are keeping Python 2 support, then start writing Python 2/3-compatible - code starting **TODAY** - + If you have dependencies that have not been ported, reach out to them to port - their project while working to make your code compatible with Python 3 so - you're ready when your dependencies are all ported - + If all your dependencies have been ported (or you have none), go ahead and - port to Python 3 +Details +======= -* If you are creating a new project that wants to have 2/3 compatibility, - code in Python 3 and then backport to Python 2 +A key point about supporting Python 2 & 3 simultaneously is that you can start +**today**! Even if your dependencies are not supporting Python 3 yet that does +not mean you can't modernize your code **now** to support Python 3. Most changes +required to support Python 3 lead to cleaner code using newer practices even in +Python 2. +Another key point is that modernizing your Python 2 code to also support +Python 3 is largely automated for you. While you might have to make some API +decisions thanks to Python 3 clarifying text data versus binary data, the +lower-level work is now mostly done for you and thus can at least benefit from +the automated changes immediately. -Before You Begin -================ +Keep those key points in mind while you read on about the details of porting +your code to support Python 2 & 3 simultaneously. -If your project is on the Cheeseshop_/PyPI_, make sure it has the proper -`trove classifiers`_ to signify what versions of Python it **currently** -supports. At minimum you should specify the major version(s), e.g. -``Programming Language :: Python :: 2`` if your project currently only supports -Python 2. It is preferable that you be as specific as possible by listing every -major/minor version of Python that you support, e.g. if your project supports -Python 2.6 and 2.7, then you want the classifiers of:: - Programming Language :: Python :: 2 - Programming Language :: Python :: 2.6 - Programming Language :: Python :: 2.7 +Drop support for Python 2.5 and older (at least) +------------------------------------------------ -Once your project supports Python 3 you will want to go back and add the -appropriate classifiers for Python 3 as well. This is important as setting the -``Programming Language :: Python :: 3`` classifier will lead to your project -being listed under the `Python 3 Packages`_ section of PyPI. +While you can make Python 2.5 work with Python 3, it is **much** easier if you +only have to work with Python 2.6 or newer (and easier still if you only have +to work with Python 2.7). If dropping Python 2.5 is not an option then the six_ +project can help you support Python 2.5 & 3 simultaneously. Do realize, though, +that nearly all the projects listed in this HOWTO will not be available to you. -Make sure you have a robust test suite. You need to -make sure everything continues to work, just like when you support a new -minor/feature release of Python. This means making sure your test suite is -thorough and is ported properly between Python 2 & 3 (consider using coverage_ -to measure that you have effective test coverage). You will also most likely -want to use something like tox_ to automate testing between all of your -supported versions of Python. You will also want to **port your tests first** so -that you can make sure that you detect breakage during the transition. Tests also -tend to be simpler than the code they are testing so it gives you an idea of how -easy it can be to port code. +If you are able to only support Python 2.6 or newer, then the required changes +to your code should continue to look and feel like idiomatic Python code. At +worst you will have to use a function instead of a method in some instances or +have to import a function instead of using a built-in one, but otherwise the +overall transformation should not feel foreign to you. -Drop support for older Python versions if possible. Python 2.5 -introduced a lot of useful syntax and libraries which have become idiomatic -in Python 3. Python 2.6 introduced future statements which makes -compatibility much easier if you are going from Python 2 to 3. -Python 2.7 continues the trend in the stdlib. Choose the newest version -of Python which you believe can be your minimum support version -and work from there. +But please aim for Python 2.7. Bugfixes for that version of Python will continue +until 2020 while Python 2.6 is no longer supported. There are also some tools +mentioned in this HOWTO which do not support Python 2.6 (e.g., Pylint_), and +this will become more commonplace as time goes on. -Target the newest version of Python 3 that you can. Beyond just the usual -bugfixes, compatibility has continued to improve between Python 2 and 3 as time -has passed. E.g. Python 3.3 added back the ``u`` prefix for -strings, making source-compatible Python code easier to write. +Make sure you specify the proper version support in your ``setup.py`` file +-------------------------------------------------------------------------- +In your ``setup.py`` file you should have the proper `trove classifier`_ +specifying what versions of Python you support. As your project does not support +Python 3 yet you should at least have +``Programming Language :: Python :: 2 :: Only`` specified. Ideally you should +also specify each major/minor version of Python that you do support, e.g. +``Programming Language :: Python :: 2.7``. -Writing Source-Compatible Python 2/3 Code -========================================= +Have good test coverage +----------------------- -Over the years the Python community has discovered that the easiest way to -support both Python 2 and 3 in parallel is to write Python code that works in -either version. While this might sound counter-intuitive at first, it actually -is not difficult and typically only requires following some select -(non-idiomatic) practices and using some key projects to help make bridging -between Python 2 and 3 easier. +Once you have your code supporting the oldest version of Python 2 you want it +to, you will want to make sure your test suite has good coverage. A good rule of +thumb is that if you want to be confident enough in your test suite that any +failures that appear after having tools rewrite your code are actual bugs in the +tools and not in your code. If you want a number to aim for, try to get over 80% +coverage (and don't feel bad if you can't easily get past 90%). If you +don't already have a tool to measure test coverage then coverage.py_ is +recommended. -Projects to Consider --------------------- +Learn the differences between Python 2 & 3 +------------------------------------------- -The lowest level library for supporting Python 2 & 3 simultaneously is six_. -Reading through its documentation will give you an idea of where exactly the -Python language changed between versions 2 & 3 and thus what you will want the -library to help you continue to support. +Once you have your code well-tested you are ready to begin porting your code to +Python 3! But to fully understand how your code is going to change and what +you want to look out for while you code, you will want to learn what changes +Python 3 makes in terms of Python 2. Typically the two best ways of doing that +is reading the `"What's New"`_ doc for each release of Python 3 and the +`Porting to Python 3`_ book (which is free online). -To help automate porting your code over to using six, you can use -modernize_. This project will attempt to rewrite your code to be as modern as -possible while using six to smooth out any differences between Python 2 & 3. -If you want to write your compatible code to feel more like Python 3 there is -the future_ project. It tries to provide backports of objects from Python 3 so -that you can use them from Python 2-compatible code, e.g. replacing the -``bytes`` type from Python 2 with the one from Python 3. -It also provides a translation script like modernize (its translation code is -actually partially based on it) to help start working with a pre-existing code -base. It is also unique in that its translation script will also port Python 3 -code backwards as well as Python 2 code forwards. +Update your code +---------------- +Once you feel like you know what is different in Python 3 compared to Python 2, +it's time to update your code! You have a choice between two tools in porting +your code automatically: Modernize_ and Futurize_. Which tool you choose will +depend on how much like Python 3 you want your code to be. Futurize_ does its +best to make Python 3 idioms and practices exist in Python 2, e.g. backporting +the ``bytes`` type from Python 3 so that you have semantic parity between the +major versions of Python. Modernize_, +on the other hand, is more conservative and targets a Python 2/3 subset of +Python, relying on six_ to help provide compatibility. -Tips & Tricks -------------- +Regardless of which tool you choose, they will update your code to run under +Python 3 while staying compatible with the version of Python 2 you started with. +Depending on how conservative you want to be, you may want to run the tool over +your test suite first and visually inspect the diff to make sure the +transformation is accurate. After you have transformed your test suite and +verified that all the tests still pass as expected, then you can transform your +application code knowing that any tests which fail is a translation failure. -To help with writing source-compatible code using one of the projects mentioned -in `Projects to Consider`_, consider following the below suggestions. Some of -them are handled by the suggested projects, so if you do use one of them then -read their documentation first to see which suggestions below will taken care of -for you. +Unfortunately the tools can't automate everything to make your code work under +Python 3 and so there are a handful of things you will need to update manually +to get full Python 3 support (which of these steps are necessary vary between +the tools). Read the documentation for the tool you choose to use to see what it +fixes by default and what it can do optionally to know what will (not) be fixed +for you and what you may have to fix on your own (e.g. using ``io.open()`` over +the built-in ``open()`` function is off by default in Modernize). Luckily, +though, there are only a couple of things to watch out for which can be +considered large issues that may be hard to debug if not watched for. -Support Python 2.7 -////////////////// +Division +++++++++ -As a first step, make sure that your project is compatible with Python 2.7. -This is just good to do as Python 2.7 is the last release of Python 2 and thus -will be used for a rather long time. It also allows for use of the ``-3`` flag -to Python to help discover places in your code where compatibility might be an -issue (the ``-3`` flag is in Python 2.6 but Python 2.7 adds more warnings). +In Python 3, ``5 / 2 == 2.5`` and not ``2``; all division between ``int`` values +result in a ``float``. This change has actually been planned since Python 2.2 +which was released in 2002. Since then users have been encouraged to add +``from __future__ import division`` to any and all files which use the ``/`` and +``//`` operators or to be running the interpreter with the ``-Q`` flag. If you +have not been doing this then you will need to go through your code and do two +things: -Try to Support Python 2.6 and Newer Only -//////////////////////////////////////// +#. Add ``from __future__ import division`` to your files +#. Update any division operator as necessary to either use ``//`` to use floor + division or continue using ``/`` and expect a float -While not possible for all projects, if you can support Python 2.6 and newer -**only**, your life will be much easier. Various future statements, stdlib -additions, etc. exist only in Python 2.6 and later which greatly assist in -supporting Python 3. But if you project must keep support for Python 2.5 then -it is still possible to simultaneously support Python 3. +The reason that ``/`` isn't simply translated to ``//`` automatically is that if +an object defines its own ``__div__`` method but not ``__floordiv__`` then your +code would begin to fail. -Below are the benefits you gain if you only have to support Python 2.6 and -newer. Some of these options are personal choice while others are -**strongly** recommended (the ones that are more for personal choice are -labeled as such). If you continue to support older versions of Python then you -at least need to watch out for situations that these solutions fix and handle -them appropriately (which is where library help from e.g. six_ comes in handy). +Text versus binary data ++++++++++++++++++++++++ +In Python 2 you could use the ``str`` type for both text and binary data. +Unfortunately this confluence of two different concepts could lead to brittle +code which sometimes worked for either kind of data, sometimes not. It also +could lead to confusing APIs if people didn't explicitly state that something +that accepted ``str`` accepted either text or binary data instead of one +specific type. This complicated the situation especially for anyone supporting +multiple languages as APIs wouldn't bother explicitly supporting ``unicode`` +when they claimed text data support. -``from __future__ import print_function`` -''''''''''''''''''''''''''''''''''''''''' +To make the distinction between text and binary data clearer and more +pronounced, Python 3 did what most languages created in the age of the internet +have done and made text and binary data distinct types that cannot blindly be +mixed together (Python predates widespread access to the internet). For any code +that only deals with text or only binary data, this separation doesn't pose an +issue. But for code that has to deal with both, it does mean you might have to +now care about when you are using text compared to binary data, which is why +this cannot be entirely automated. -It will not only get you used to typing ``print()`` as a function instead of a -statement, but it will also give you the various benefits the function has over -the Python 2 statement (six_ provides a function if you support Python 2.5 or -older). +To start, you will need to decide which APIs take text and which take binary +(it is **highly** recommended you don't design APIs that can take both due to +the difficulty of keeping the code working; as stated earlier it is difficult to +do well). In Python 2 this means making sure the APIs that take text can work +with ``unicode`` in Python 2 and those that work with binary data work with the +``bytes`` type from Python 3 and thus a subset of ``str`` in Python 2 (which the +``bytes`` type in Python 2 is an alias for). Usually the biggest issue is +realizing which methods exist for which types in Python 2 & 3 simultaneously +(for text that's ``unicode`` in Python 2 and ``str`` in Python 3, for binary +that's ``str``/``bytes`` in Python 2 and ``bytes`` in Python 3). The following +table lists the **unique** methods of each data type across Python 2 & 3 +(e.g., the ``decode()`` method is usable on the equivalent binary data type in +either Python 2 or 3, but it can't be used by the text data type consistently +between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). +======================== ===================== +**Text data** **Binary data** +------------------------ --------------------- +__mod__ (``%`` operator) +------------------------ --------------------- +\ decode +------------------------ --------------------- +encode +------------------------ --------------------- +format +------------------------ --------------------- +isdecimal +------------------------ --------------------- +isnumeric +======================== ===================== -``from __future__ import unicode_literals`` -''''''''''''''''''''''''''''''''''''''''''' +Making the distinction easier to handle can be accomplished by encoding and +decoding between binary data and text at the edge of your code. This means that +when you receive text in binary data, you should immediately decode it. And if +your code needs to send text as binary data then encode it as late as possible. +This allows your code to work with only text internally and thus eliminates +having to keep track of what type of data you are working with. -If you choose to use this future statement then all string literals in -Python 2 will be assumed to be Unicode (as is already the case in Python 3). -If you choose not to use this future statement then you should mark all of your -text strings with a ``u`` prefix and only support Python 3.3 or newer. But you -are **strongly** advised to do one or the other (six_ provides a function in -case you don't want to use the future statement **and** you want to support -Python 3.2 or older). +The next issue is making sure you know whether the string literals in your code +represent text or binary data. At minimum you should add a ``b`` prefix to any +literal that presents binary data. For text you should either use the +``from __future__ import unicode_literals`` statement or add a ``u`` prefix to +the text literal. - -Bytes/string literals -''''''''''''''''''''' - -This is a **very** important one. Prefix Python 2 strings that -are meant to contain bytes with a ``b`` prefix to very clearly delineate -what is and is not a Python 3 text string (six_ provides a function to use for -Python 2.5 compatibility). - -This point cannot be stressed enough: make sure you know what all of your string -literals in Python 2 are meant to be in Python 3. Any string literal that -should be treated as bytes should have the ``b`` prefix. Any string literal -that should be Unicode/text in Python 2 should either have the ``u`` literal -(supported, but ignored, in Python 3.3 and later) or you should have -``from __future__ import unicode_literals`` at the top of the file. But the key -point is you should know how Python 3 will treat every one one of your string -literals and you should mark them as appropriate. - -There are some differences between byte literals in Python 2 and those in -Python 3 thanks to the bytes type just being an alias to ``str`` in Python 2. -See the `Handle Common "Gotchas"`_ section for what to watch out for. - -``from __future__ import absolute_import`` -'''''''''''''''''''''''''''''''''''''''''' -Discussed in more detail below, but you should use this future statement to -prevent yourself from accidentally using implicit relative imports. - - -Supporting Python 2.5 and Newer Only -//////////////////////////////////// - -If you are supporting Python 2.5 and newer there are still some features of -Python that you can utilize. - - -``from __future__ import absolute_import`` -'''''''''''''''''''''''''''''''''''''''''' - -Implicit relative imports (e.g., importing ``spam.bacon`` from within -``spam.eggs`` with the statement ``import bacon``) do not work in Python 3. -This future statement moves away from that and allows the use of explicit -relative imports (e.g., ``from . import bacon``). - -In Python 2.5 you must use -the __future__ statement to get to use explicit relative imports and prevent -implicit ones. In Python 2.6 explicit relative imports are available without -the statement, but you still want the __future__ statement to prevent implicit -relative imports. In Python 2.7 the __future__ statement is not needed. In -other words, unless you are only supporting Python 2.7 or a version earlier -than Python 2.5, use this __future__ statement. - - -Mark all Unicode strings with a ``u`` prefix -''''''''''''''''''''''''''''''''''''''''''''' - -While Python 2.6 has a ``__future__`` statement to automatically cause Python 2 -to treat all string literals as Unicode, Python 2.5 does not have that shortcut. -This means you should go through and mark all string literals with a ``u`` -prefix to turn them explicitly into text strings where appropriate and only -support Python 3.3 or newer. Otherwise use a project like six_ which provides a -function to pass all text string literals through. - - -Capturing the Currently Raised Exception -'''''''''''''''''''''''''''''''''''''''' - -In Python 2.5 and earlier the syntax to access the current exception is:: - - try: - raise Exception() - except Exception, exc: - # Current exception is 'exc'. - pass - -This syntax changed in Python 3 (and backported to Python 2.6 and later) -to:: - - try: - raise Exception() - except Exception as exc: - # Current exception is 'exc'. - # In Python 3, 'exc' is restricted to the block; in Python 2.6/2.7 it will "leak". - pass - -Because of this syntax change you must change how you capture the current -exception in Python 2.5 and earlier to:: - - try: - raise Exception() - except Exception: - import sys - exc = sys.exc_info()[1] - # Current exception is 'exc'. - pass - -You can get more information about the raised exception from -:func:`sys.exc_info` than simply the current exception instance, but you most -likely don't need it. - -.. note:: - In Python 3, the traceback is attached to the exception instance - through the ``__traceback__`` attribute. If the instance is saved in - a local variable that persists outside of the ``except`` block, the - traceback will create a reference cycle with the current frame and its - dictionary of local variables. This will delay reclaiming dead - resources until the next cyclic :term:`garbage collection` pass. - - In Python 2, this problem only occurs if you save the traceback itself - (e.g. the third element of the tuple returned by :func:`sys.exc_info`) - in a variable. - - -Handle Common "Gotchas" -/////////////////////// - -These are things to watch out for no matter what version of Python 2 you are -supporting which are not syntactic considerations. - - -``from __future__ import division`` -''''''''''''''''''''''''''''''''''' - -While the exact same outcome can be had by using the ``-Qnew`` argument to -Python, using this future statement lifts the requirement that your users use -the flag to get the expected behavior of division in Python 3 -(e.g., ``1/2 == 0.5; 1//2 == 0``). - - - -Specify when opening a file as binary -''''''''''''''''''''''''''''''''''''' - +As part of this dichotomy you also need to be careful about opening files. Unless you have been working on Windows, there is a chance you have not always bothered to add the ``b`` mode when opening a binary file (e.g., ``rb`` for binary reading). Under Python 3, binary files and text files are clearly distinct and mutually incompatible; see the :mod:`io` module for details. Therefore, you **must** make a decision of whether a file will be used for -binary access (allowing to read and/or write bytes data) or text access -(allowing to read and/or write unicode data). +binary access (allowing to read and/or write binary data) or text access +(allowing to read and/or write text data). You should also use :func:`io.open` +for opening files instead of the built-in :func:`open` function as the :mod:`io` +module is consistent from Python 2 to 3 while the built-in :func:`open` function +is not (in Python 3 it's actually :func:`io.open`). -Text files -'''''''''' +Finally, the indexing of binary data requires careful handling (slicing does +**not** require any special handling). In Python 2, +``b'123'[1] == b'2'`` while in Python 3 ``b'123'[1] == 50``. Because binary data +is simply a collection of binary numbers, Python 3 returns the integer value for +the byte you index on. But in Python 2 because ``bytes == str``, indexing +returns a one-item slice of bytes. The six_ project has a function +named ``six.indexbytes()`` which will return an integer like in Python 3: +``six.indexbytes(b'123', 1)``. -Text files created using ``open()`` under Python 2 return byte strings, -while under Python 3 they return unicode strings. Depending on your porting -strategy, this can be an issue. +To summarize: -If you want text files to return unicode strings in Python 2, you have two -possibilities: +#. Decide which of your APIs take text and which take binary data +#. Make sure that your code that works with text also works with ``unicode`` and + code for binary data works with ``bytes`` in Python 2 (see the table above + for what methods you cannot use for each type) +#. Mark all binary literals with a ``b`` prefix, use a ``u`` prefix or + :mod:`__future__` import statement for text literals +#. Decode binary data to text as soon as possible, encode text as binary data as + late as possible +#. Open files using :func:`io.open` and make sure to specify the ``b`` mode when + appropriate +#. Be careful when indexing binary data -* Under Python 2.6 and higher, use :func:`io.open`. Since :func:`io.open` - is essentially the same function in both Python 2 and Python 3, it will - help iron out any issues that might arise. +Prevent compatibility regressions +--------------------------------- -* If pre-2.6 compatibility is needed, then you should use :func:`codecs.open` - instead. This will make sure that you get back unicode strings in Python 2. +Once you have fully translated your code to be compatible with Python 3, you +will want to make sure your code doesn't regress and stop working under +Python 3. This is especially true if you have a dependency which is blocking you +from actually running under Python 3 at the moment. -Subclass ``object`` -''''''''''''''''''' +To help with staying compatible, any new modules you create should have +at least the following block of code at the top of it:: -New-style classes have been around since Python 2.2. You need to make sure -you are subclassing from ``object`` to avoid odd edge cases involving method -resolution order, etc. This continues to be totally valid in Python 3 (although -unneeded as all classes implicitly inherit from ``object``). + from __future__ import absolute_import + from __future__ import division + from __future__ import print_statement + from __future__ import unicode_literals +You can also run Python 2 with the ``-3`` flag to be warned about various +compatibility issues your code triggers during execution. If you turn warnings +into errors with ``-Werror`` then you can make sure that you don't accidentally +miss a warning. -Deal With the Bytes/String Dichotomy -'''''''''''''''''''''''''''''''''''' -One of the biggest issues people have when porting code to Python 3 is handling -the bytes/string dichotomy. Because Python 2 allowed the ``str`` type to hold -textual data, people have over the years been rather loose in their delineation -of what ``str`` instances held text compared to bytes. In Python 3 you cannot -be so care-free anymore and need to properly handle the difference. The key to -handling this issue is to make sure that **every** string literal in your -Python 2 code is either syntactically or functionally marked as either bytes or -text data. After this is done you then need to make sure your APIs are designed -to either handle a specific type or made to be properly polymorphic. +You can also use the Pylint_ project and its ``--py3k`` flag to lint your code +to receive warnings when your code begins to deviate from Python 3 +compatibility. This also prevents you from having to run Modernize_ or Futurize_ +over your code regularly to catch compatibility regressions. This does require +you only support Python 2.7 and Python 3.4 or newer as that is Pylint's +minimum Python version support. -Mark Up Python 2 String Literals -******************************** +Check which dependencies block your transition +---------------------------------------------- -First thing you must do is designate every single string literal in Python 2 -as either textual or bytes data. If you are only supporting Python 2.6 or -newer, this can be accomplished by marking bytes literals with a ``b`` prefix -and then designating textual data with a ``u`` prefix or using the -``unicode_literals`` future statement. +**After** you have made your code compatible with Python 3 you should begin to +care about whether your dependencies have also been ported. The caniusepython3_ +project was created to help you determine which projects +-- directly or indirectly -- are blocking you from supporting Python 3. There +is both a command-line tool as well as a web interface at +https://caniusepython3.com . -If your project supports versions of Python predating 2.6, then you should use -the six_ project and its ``b()`` function to denote bytes literals. For text -literals you can either use six's ``u()`` function or use a ``u`` prefix. +The project also provides code which you can integrate into your test suite so +that you will have a failing test when you no longer have dependencies blocking +you from using Python 3. This allows you to avoid having to manually check your +dependencies and to be notified quickly when you can start running on Python 3. +Update your ``setup.py`` file to denote Python 3 compatibility +-------------------------------------------------------------- -Decide what APIs Will Accept -**************************** +Once your code works under Python 3, you should update the classifiers in +your ``setup.py`` to contain ``Programming Language :: Python :: 3`` and to not +specify sole Python 2 support. This will tell +anyone using your code that you support Python 2 **and** 3. Ideally you will +also want to add classifiers for each major/minor version of Python you now +support. -In Python 2 it was very easy to accidentally create an API that accepted both -bytes and textual data. But in Python 3, thanks to the more strict handling of -disparate types, this loose usage of bytes and text together tends to fail. +Use continuous integration to stay compatible +--------------------------------------------- -Take the dict ``{b'a': 'bytes', u'a': 'text'}`` in Python 2.6. It creates the -dict ``{u'a': 'text'}`` since ``b'a' == u'a'``. But in Python 3 the equivalent -dict creates ``{b'a': 'bytes', 'a': 'text'}``, i.e., no lost data. Similar -issues can crop up when transitioning Python 2 code to Python 3. +Once you are able to fully run under Python 3 you will want to make sure your +code always works under both Python 2 & 3. Probably the best tool for running +your tests under multiple Python interpreters is tox_. You can then integrate +tox with your continuous integration system so that you never accidentally break +Python 2 or 3 support. -This means you need to choose what an API is going to accept and create and -consistently stick to that API in both Python 2 and 3. +You may also want to use use the ``-bb`` flag with the Python 3 interpreter to +trigger an exception when you are comparing bytes to strings. Usually it's +simply ``False``, but if you made a mistake in your separation of text/binary +data handling you may be accidentally comparing text and binary data. This flag +will raise an exception when that occurs to help track down such cases. +And that's mostly it! At this point your code base is compatible with both +Python 2 and 3 simultaneously. Your testing will also be set up so that you +don't accidentally break Python 2 or 3 compatibility regardless of which version +you typically run your tests under while developing. -Bytes / Unicode Comparison -************************** -In Python 3, mixing bytes and unicode is forbidden in most situations; it -will raise a :class:`TypeError` where Python 2 would have attempted an implicit -coercion between types. However, there is one case where it doesn't and -it can be very misleading:: +Dropping Python 2 support completely +==================================== - >>> b"" == "" - False +If you are able to fully drop support for Python 2, then the steps required +to transition to Python 3 simplify greatly. -This is because an equality comparison is required by the language to always -succeed (and return ``False`` for incompatible types). However, this also -means that code incorrectly ported to Python 3 can display buggy behaviour -if such comparisons are silently executed. To detect such situations, -Python 3 has a ``-b`` flag that will display a warning:: +#. Update your code to only support Python 2.7 +#. Make sure you have good test coverage (coverage.py_ can help) +#. Learn the differences between Python 2 & 3 +#. Use 2to3_ to rewrite your code to run only under Python 3 - $ python3 -b - >>> b"" == "" - __main__:1: BytesWarning: Comparison between bytes and string - False +After this your code will be fully Python 3 compliant but in a way that is not +supported by Python 2. You should also update the classifiers in your +``setup.py`` to contain ``Programming Language :: Python :: 3 :: Only``. -To turn the warning into an exception, use the ``-bb`` flag instead:: - $ python3 -bb - >>> b"" == "" - Traceback (most recent call last): - File "", line 1, in - BytesWarning: Comparison between bytes and string +.. _2to3: https://docs.python.org/3/library/2to3.html +.. _caniusepython3: +.. _coverage.py: https://pypi.python.org/pypi/coverage +.. _Futurize: http://python-future.org/automatic_conversion.html +.. _Modernize: +.. _Porting to Python 3: http://python3porting.com/ +.. _Pylint: https://pypi.python.org/pypi/pylint +.. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html - -Indexing bytes objects -'''''''''''''''''''''' - -Another potentially surprising change is the indexing behaviour of bytes -objects in Python 3:: - - >>> b"xyz"[0] - 120 - -Indeed, Python 3 bytes objects (as well as :class:`bytearray` objects) -are sequences of integers. But code converted from Python 2 will often -assume that indexing a bytestring produces another bytestring, not an -integer. To reconcile both behaviours, use slicing:: - - >>> b"xyz"[0:1] - b'x' - >>> n = 1 - >>> b"xyz"[n:n+1] - b'y' - -The only remaining gotcha is that an out-of-bounds slice returns an empty -bytes object instead of raising ``IndexError``: - - >>> b"xyz"[3] - Traceback (most recent call last): - File "", line 1, in - IndexError: index out of range - >>> b"xyz"[3:4] - b'' - - -``__str__()``/``__unicode__()`` -''''''''''''''''''''''''''''''' - -In Python 2, objects can specify both a string and unicode representation of -themselves. In Python 3, though, there is only a string representation. This -becomes an issue as people can inadvertently do things in their ``__str__()`` -methods which have unpredictable results (e.g., infinite recursion if you -happen to use the ``unicode(self).encode('utf8')`` idiom as the body of your -``__str__()`` method). - -You can use a mixin class to work around this. This allows you to only define a -``__unicode__()`` method for your class and let the mixin derive -``__str__()`` for you (code from -http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/):: - - import sys - - class UnicodeMixin(object): - - """Mixin class to handle defining the proper __str__/__unicode__ - methods in Python 2 or 3.""" - - if sys.version_info[0] >= 3: # Python 3 - def __str__(self): - return self.__unicode__() - else: # Python 2 - def __str__(self): - return self.__unicode__().encode('utf8') - - - class Spam(UnicodeMixin): - - def __unicode__(self): - return u'spam-spam-bacon-spam' # 2to3 will remove the 'u' prefix - - -Don't Index on Exceptions -''''''''''''''''''''''''' - -In Python 2, the following worked:: - - >>> exc = Exception(1, 2, 3) - >>> exc.args[1] - 2 - >>> exc[1] # Python 2 only! - 2 - -But in Python 3, indexing directly on an exception is an error. You need to -make sure to only index on the :attr:`BaseException.args` attribute which is a -sequence containing all arguments passed to the :meth:`__init__` method. - -Even better is to use the documented attributes the exception provides. - - -Don't use ``__getslice__`` & Friends -'''''''''''''''''''''''''''''''''''' - -Been deprecated for a while, but Python 3 finally drops support for -``__getslice__()``, etc. Move completely over to :meth:`__getitem__` and -friends. - - -Updating doctests -''''''''''''''''' - -Don't forget to make them Python 2/3 compatible as well. If you wrote a -monolithic set of doctests (e.g., a single docstring containing all of your -doctests), you should at least consider breaking the doctests up into smaller -pieces to make it more manageable to fix. Otherwise it might very well be worth -your time and effort to port your tests to :mod:`unittest`. - - -Update ``map`` for imbalanced input sequences -''''''''''''''''''''''''''''''''''''''''''''' - -With Python 2, when ``map`` was given more than one input sequence it would pad -the shorter sequences with ``None`` values, returning a sequence as long as the -longest input sequence. - -With Python 3, if the input sequences to ``map`` are of unequal length, ``map`` -will stop at the termination of the shortest of the sequences. For full -compatibility with ``map`` from Python 2.x, wrap the sequence arguments in -:func:`itertools.zip_longest`, e.g. ``map(func, *sequences)`` becomes -``list(map(func, itertools.zip_longest(*sequences)))``. - -Eliminate ``-3`` Warnings -------------------------- - -When you run your application's test suite, run it using the ``-3`` flag passed -to Python. This will cause various warnings to be raised during execution about -things that are semantic changes between Python 2 and 3. Try to eliminate those -warnings to make your code even more portable to Python 3. - - -Alternative Approaches -====================== - -While supporting Python 2 & 3 simultaneously is typically the preferred choice -by people so that they can continue to improve code and have it work for the -most number of users, your life may be easier if you only have to support one -major version of Python going forward. - -Supporting Only Python 3 Going Forward From Python 2 Code ---------------------------------------------------------- - -If you have Python 2 code but going forward only want to improve it as Python 3 -code, then you can use :ref:`2to3 <2to3-reference>` to translate your Python 2 -code to Python 3 code. This is only recommended, though, if your current -version of your project is going into maintenance mode and you want all new -features to be exclusive to Python 3. - - -Backporting Python 3 code to Python 2 -------------------------------------- - -If you have Python 3 code and have little interest in supporting Python 2 you -can use 3to2_ to translate from Python 3 code to Python 2 code. This is only -recommended if you don't plan to heavily support Python 2 users. Otherwise -write your code for Python 3 and then backport as far back as you want. This -is typically easier than going from Python 2 to 3 as you will have worked out -any difficulties with e.g. bytes/strings, etc. - - -Other Resources -=============== - -The authors of the following blog posts, wiki pages, and books deserve special -thanks for making public their tips for porting Python 2 code to Python 3 (and -thus helping provide information for this document and its various revisions -over the years): - -* https://wiki.python.org/moin/PortingPythonToPy3k -* http://python3porting.com/ -* http://docs.pythonsprints.com/python3_porting/py-porting.html -* http://techspot.zzzeek.org/2011/01/24/zzzeek-s-guide-to-python-3-porting/ -* http://dabeaz.blogspot.com/2011/01/porting-py65-and-my-superboard-to.html -* http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ -* http://lucumr.pocoo.org/2010/2/11/porting-to-python-3-a-guide/ -* https://wiki.ubuntu.com/Python/3 - -If you feel there is something missing from this document that should be added, -please email the python-porting_ mailing list. - - -.. _3to2: https://pypi.python.org/pypi/3to2 -.. _Cheeseshop: PyPI_ -.. _coverage: https://pypi.python.org/pypi/coverage -.. _future: http://python-future.org/ -.. _modernize: https://github.com/mitsuhiko/python-modernize -.. _Porting to Python 3: http://python3porting.com/ -.. _PyPI: https://pypi.python.org/pypi -.. _Python 3 Packages: https://pypi.python.org/pypi?:action=browse&c=533&show=all -.. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html +.. _python-future: http://python-future.org/ .. _python-porting: https://mail.python.org/mailman/listinfo/python-porting .. _six: https://pypi.python.org/pypi/six .. _tox: https://pypi.python.org/pypi/tox -.. _trove classifiers: https://pypi.python.org/pypi?%3Aaction=list_classifiers +.. _trove classifier: https://pypi.python.org/pypi?%3Aaction=list_classifiers +.. _"What's New": https://docs.python.org/3/whatsnew/index.html -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 17:02:07 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 05 Dec 2014 16:02:07 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyOTE0?= =?utf-8?q?=3A_Update_the_Python_2/3_porting_HOWTO_to_describe_a_more?= Message-ID: <20141205160143.76716.58966@psf.io> https://hg.python.org/cpython/rev/55b94462ca7f changeset: 93737:55b94462ca7f branch: 3.4 parent: 93735:cd3244416592 user: Brett Cannon date: Fri Dec 05 10:56:12 2014 -0500 summary: Issue #22914: Update the Python 2/3 porting HOWTO to describe a more automated process. files: Doc/howto/pyporting.rst | 803 +++++++++------------------ Misc/NEWS | 3 + 2 files changed, 284 insertions(+), 522 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -19,600 +19,359 @@ If you would like to read one core Python developer's take on why Python 3 came into existence, you can read Nick Coghlan's `Python 3 Q & A`_. - If you prefer to read a (free) book on porting a project to Python 3, - consider reading `Porting to Python 3`_ by Lennart Regebro which should cover - much of what is discussed in this HOWTO. - For help with porting, you can email the python-porting_ mailing list with questions. -The Short Version -================= +The Short Explanation +===================== -* Decide what's the oldest version of Python 2 you want to support (if at all) -* Make sure you have a thorough test suite and use continuous integration - testing to make sure you stay compatible with the versions of Python you care - about -* If you have dependencies, check their Python 3 status using caniusepython3 - (`command-line tool `__, - `web app `__) +To make your project be single-source Python 2/3 compatible, the basic steps +are: -With that done, your options are: +#. Update your code to drop support for Python 2.5 or older (supporting only + Python 2.7 is ideal) +#. Make sure you have good test coverage (coverage.py_ can help) +#. Learn the differences between Python 2 & 3 +#. Use Modernize_ or Futurize_ to update your code +#. Use Pylint_ to help make sure you don't regress on your Python 3 support + (if only supporting Python 2.7/3.4 or newer) +#. Use caniusepython3_ to find out which of your dependencies are blocking your + use of Python 3 +#. Once your dependencies are no longer blocking you, use continuous integration + to make sure you stay compatible with Python 2 & 3 (tox_ can help test + against multiple versions of Python) -* If you are dropping Python 2 support, use :ref:`2to3 <2to3-reference>` to port - to Python 3 +If you are dropping support for Python 2 entirely, then after you learn the +differences between Python 2 & 3 you can run 2to3_ over your code and skip the +rest of the steps outlined above. -* If you are keeping Python 2 support, then start writing Python 2/3-compatible - code starting **TODAY** - + If you have dependencies that have not been ported, reach out to them to port - their project while working to make your code compatible with Python 3 so - you're ready when your dependencies are all ported - + If all your dependencies have been ported (or you have none), go ahead and - port to Python 3 +Details +======= -* If you are creating a new project that wants to have 2/3 compatibility, - code in Python 3 and then backport to Python 2 +A key point about supporting Python 2 & 3 simultaneously is that you can start +**today**! Even if your dependencies are not supporting Python 3 yet that does +not mean you can't modernize your code **now** to support Python 3. Most changes +required to support Python 3 lead to cleaner code using newer practices even in +Python 2. +Another key point is that modernizing your Python 2 code to also support +Python 3 is largely automated for you. While you might have to make some API +decisions thanks to Python 3 clarifying text data versus binary data, the +lower-level work is now mostly done for you and thus can at least benefit from +the automated changes immediately. -Before You Begin -================ +Keep those key points in mind while you read on about the details of porting +your code to support Python 2 & 3 simultaneously. -If your project is on the Cheeseshop_/PyPI_, make sure it has the proper -`trove classifiers`_ to signify what versions of Python it **currently** -supports. At minimum you should specify the major version(s), e.g. -``Programming Language :: Python :: 2`` if your project currently only supports -Python 2. It is preferrable that you be as specific as possible by listing every -major/minor version of Python that you support, e.g. if your project supports -Python 2.6 and 2.7, then you want the classifiers of:: - Programming Language :: Python :: 2 - Programming Language :: Python :: 2.6 - Programming Language :: Python :: 2.7 +Drop support for Python 2.5 and older (at least) +------------------------------------------------ -Once your project supports Python 3 you will want to go back and add the -appropriate classifiers for Python 3 as well. This is important as setting the -``Programming Language :: Python :: 3`` classifier will lead to your project -being listed under the `Python 3 Packages`_ section of PyPI. +While you can make Python 2.5 work with Python 3, it is **much** easier if you +only have to work with Python 2.6 or newer (and easier still if you only have +to work with Python 2.7). If dropping Python 2.5 is not an option then the six_ +project can help you support Python 2.5 & 3 simultaneously. Do realize, though, +that nearly all the projects listed in this HOWTO will not be available to you. -Make sure you have a robust test suite. You need to -make sure everything continues to work, just like when you support a new -minor/feature release of Python. This means making sure your test suite is -thorough and is ported properly between Python 2 & 3 (consider using coverage_ -to measure that you have effective test coverage). You will also most likely -want to use something like tox_ to automate testing between all of your -supported versions of Python. You will also want to **port your tests first** so -that you can make sure that you detect breakage during the transition. Tests also -tend to be simpler than the code they are testing so it gives you an idea of how -easy it can be to port code. +If you are able to only support Python 2.6 or newer, then the required changes +to your code should continue to look and feel like idiomatic Python code. At +worst you will have to use a function instead of a method in some instances or +have to import a function instead of using a built-in one, but otherwise the +overall transformation should not feel foreign to you. -Drop support for older Python versions if possible. Python 2.5 -introduced a lot of useful syntax and libraries which have become idiomatic -in Python 3. Python 2.6 introduced future statements which makes -compatibility much easier if you are going from Python 2 to 3. -Python 2.7 continues the trend in the stdlib. Choose the newest version -of Python which you believe can be your minimum support version -and work from there. +But please aim for Python 2.7. Bugfixes for that version of Python will continue +until 2020 while Python 2.6 is no longer supported. There are also some tools +mentioned in this HOWTO which do not support Python 2.6 (e.g., Pylint_), and +this will become more commonplace as time goes on. -Target the newest version of Python 3 that you can. Beyond just the usual -bugfixes, compatibility has continued to improve between Python 2 and 3 as time -has passed. E.g. Python 3.3 added back the ``u`` prefix for -strings, making source-compatible Python code easier to write. +Make sure you specify the proper version support in your ``setup.py`` file +-------------------------------------------------------------------------- +In your ``setup.py`` file you should have the proper `trove classifier`_ +specifying what versions of Python you support. As your project does not support +Python 3 yet you should at least have +``Programming Language :: Python :: 2 :: Only`` specified. Ideally you should +also specify each major/minor version of Python that you do support, e.g. +``Programming Language :: Python :: 2.7``. -Writing Source-Compatible Python 2/3 Code -========================================= +Have good test coverage +----------------------- -Over the years the Python community has discovered that the easiest way to -support both Python 2 and 3 in parallel is to write Python code that works in -either version. While this might sound counter-intuitive at first, it actually -is not difficult and typically only requires following some select -(non-idiomatic) practices and using some key projects to help make bridging -between Python 2 and 3 easier. +Once you have your code supporting the oldest version of Python 2 you want it +to, you will want to make sure your test suite has good coverage. A good rule of +thumb is that if you want to be confident enough in your test suite that any +failures that appear after having tools rewrite your code are actual bugs in the +tools and not in your code. If you want a number to aim for, try to get over 80% +coverage (and don't feel bad if you can't easily get past 90%). If you +don't already have a tool to measure test coverage then coverage.py_ is +recommended. -Projects to Consider --------------------- +Learn the differences between Python 2 & 3 +------------------------------------------- -The lowest level library for supporting Python 2 & 3 simultaneously is six_. -Reading through its documentation will give you an idea of where exactly the -Python language changed between versions 2 & 3 and thus what you will want the -library to help you continue to support. +Once you have your code well-tested you are ready to begin porting your code to +Python 3! But to fully understand how your code is going to change and what +you want to look out for while you code, you will want to learn what changes +Python 3 makes in terms of Python 2. Typically the two best ways of doing that +is reading the `"What's New"`_ doc for each release of Python 3 and the +`Porting to Python 3`_ book (which is free online). -To help automate porting your code over to using six, you can use -modernize_. This project will attempt to rewrite your code to be as modern as -possible while using six to smooth out any differences between Python 2 & 3. -If you want to write your compatible code to feel more like Python 3 there is -the future_ project. It tries to provide backports of objects from Python 3 so -that you can use them from Python 2-compatible code, e.g. replacing the -``bytes`` type from Python 2 with the one from Python 3. -It also provides a translation script like modernize (its translation code is -actually partially based on it) to help start working with a pre-existing code -base. It is also unique in that its translation script will also port Python 3 -code backwards as well as Python 2 code forwards. +Update your code +---------------- +Once you feel like you know what is different in Python 3 compared to Python 2, +it's time to update your code! You have a choice between two tools in porting +your code automatically: Modernize_ and Futurize_. Which tool you choose will +depend on how much like Python 3 you want your code to be. Futurize_ does its +best to make Python 3 idioms and practices exist in Python 2, e.g. backporting +the ``bytes`` type from Python 3 so that you have semantic parity between the +major versions of Python. Modernize_, +on the other hand, is more conservative and targets a Python 2/3 subset of +Python, relying on six_ to help provide compatibility. -Tips & Tricks -------------- +Regardless of which tool you choose, they will update your code to run under +Python 3 while staying compatible with the version of Python 2 you started with. +Depending on how conservative you want to be, you may want to run the tool over +your test suite first and visually inspect the diff to make sure the +transformation is accurate. After you have transformed your test suite and +verified that all the tests still pass as expected, then you can transform your +application code knowing that any tests which fail is a translation failure. -To help with writing source-compatible code using one of the projects mentioned -in `Projects to Consider`_, consider following the below suggestions. Some of -them are handled by the suggested projects, so if you do use one of them then -read their documentation first to see which suggestions below will taken care of -for you. +Unfortunately the tools can't automate everything to make your code work under +Python 3 and so there are a handful of things you will need to update manually +to get full Python 3 support (which of these steps are necessary vary between +the tools). Read the documentation for the tool you choose to use to see what it +fixes by default and what it can do optionally to know what will (not) be fixed +for you and what you may have to fix on your own (e.g. using ``io.open()`` over +the built-in ``open()`` function is off by default in Modernize). Luckily, +though, there are only a couple of things to watch out for which can be +considered large issues that may be hard to debug if not watched for. -Support Python 2.7 -////////////////// +Division +++++++++ -As a first step, make sure that your project is compatible with Python 2.7. -This is just good to do as Python 2.7 is the last release of Python 2 and thus -will be used for a rather long time. It also allows for use of the ``-3`` flag -to Python to help discover places in your code where compatibility might be an -issue (the ``-3`` flag is in Python 2.6 but Python 2.7 adds more warnings). +In Python 3, ``5 / 2 == 2.5`` and not ``2``; all division between ``int`` values +result in a ``float``. This change has actually been planned since Python 2.2 +which was released in 2002. Since then users have been encouraged to add +``from __future__ import division`` to any and all files which use the ``/`` and +``//`` operators or to be running the interpreter with the ``-Q`` flag. If you +have not been doing this then you will need to go through your code and do two +things: -Try to Support Python 2.6 and Newer Only -//////////////////////////////////////// +#. Add ``from __future__ import division`` to your files +#. Update any division operator as necessary to either use ``//`` to use floor + division or continue using ``/`` and expect a float -While not possible for all projects, if you can support Python 2.6 and newer -**only**, your life will be much easier. Various future statements, stdlib -additions, etc. exist only in Python 2.6 and later which greatly assist in -supporting Python 3. But if you project must keep support for Python 2.5 then -it is still possible to simultaneously support Python 3. +The reason that ``/`` isn't simply translated to ``//`` automatically is that if +an object defines its own ``__div__`` method but not ``__floordiv__`` then your +code would begin to fail. -Below are the benefits you gain if you only have to support Python 2.6 and -newer. Some of these options are personal choice while others are -**strongly** recommended (the ones that are more for personal choice are -labeled as such). If you continue to support older versions of Python then you -at least need to watch out for situations that these solutions fix and handle -them appropriately (which is where library help from e.g. six_ comes in handy). +Text versus binary data ++++++++++++++++++++++++ +In Python 2 you could use the ``str`` type for both text and binary data. +Unfortunately this confluence of two different concepts could lead to brittle +code which sometimes worked for either kind of data, sometimes not. It also +could lead to confusing APIs if people didn't explicitly state that something +that accepted ``str`` accepted either text or binary data instead of one +specific type. This complicated the situation especially for anyone supporting +multiple languages as APIs wouldn't bother explicitly supporting ``unicode`` +when they claimed text data support. -``from __future__ import print_function`` -''''''''''''''''''''''''''''''''''''''''' +To make the distinction between text and binary data clearer and more +pronounced, Python 3 did what most languages created in the age of the internet +have done and made text and binary data distinct types that cannot blindly be +mixed together (Python predates widespread access to the internet). For any code +that only deals with text or only binary data, this separation doesn't pose an +issue. But for code that has to deal with both, it does mean you might have to +now care about when you are using text compared to binary data, which is why +this cannot be entirely automated. -It will not only get you used to typing ``print()`` as a function instead of a -statement, but it will also give you the various benefits the function has over -the Python 2 statement (six_ provides a function if you support Python 2.5 or -older). +To start, you will need to decide which APIs take text and which take binary +(it is **highly** recommended you don't design APIs that can take both due to +the difficulty of keeping the code working; as stated earlier it is difficult to +do well). In Python 2 this means making sure the APIs that take text can work +with ``unicode`` in Python 2 and those that work with binary data work with the +``bytes`` type from Python 3 and thus a subset of ``str`` in Python 2 (which the +``bytes`` type in Python 2 is an alias for). Usually the biggest issue is +realizing which methods exist for which types in Python 2 & 3 simultaneously +(for text that's ``unicode`` in Python 2 and ``str`` in Python 3, for binary +that's ``str``/``bytes`` in Python 2 and ``bytes`` in Python 3). The following +table lists the **unique** methods of each data type across Python 2 & 3 +(e.g., the ``decode()`` method is usable on the equivalent binary data type in +either Python 2 or 3, but it can't be used by the text data type consistently +between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). +======================== ===================== +**Text data** **Binary data** +------------------------ --------------------- +__mod__ (``%`` operator) +------------------------ --------------------- +\ decode +------------------------ --------------------- +encode +------------------------ --------------------- +format +------------------------ --------------------- +isdecimal +------------------------ --------------------- +isnumeric +======================== ===================== -``from __future__ import unicode_literals`` -''''''''''''''''''''''''''''''''''''''''''' +Making the distinction easier to handle can be accomplished by encoding and +decoding between binary data and text at the edge of your code. This means that +when you receive text in binary data, you should immediately decode it. And if +your code needs to send text as binary data then encode it as late as possible. +This allows your code to work with only text internally and thus eliminates +having to keep track of what type of data you are working with. -If you choose to use this future statement then all string literals in -Python 2 will be assumed to be Unicode (as is already the case in Python 3). -If you choose not to use this future statement then you should mark all of your -text strings with a ``u`` prefix and only support Python 3.3 or newer. But you -are **strongly** advised to do one or the other (six_ provides a function in -case you don't want to use the future statement **and** you want to support -Python 3.2 or older). +The next issue is making sure you know whether the string literals in your code +represent text or binary data. At minimum you should add a ``b`` prefix to any +literal that presents binary data. For text you should either use the +``from __future__ import unicode_literals`` statement or add a ``u`` prefix to +the text literal. - -Bytes/string literals -''''''''''''''''''''' - -This is a **very** important one. Prefix Python 2 strings that -are meant to contain bytes with a ``b`` prefix to very clearly delineate -what is and is not a Python 3 text string (six_ provides a function to use for -Python 2.5 compatibility). - -This point cannot be stressed enough: make sure you know what all of your string -literals in Python 2 are meant to be in Python 3. Any string literal that -should be treated as bytes should have the ``b`` prefix. Any string literal -that should be Unicode/text in Python 2 should either have the ``u`` literal -(supported, but ignored, in Python 3.3 and later) or you should have -``from __future__ import unicode_literals`` at the top of the file. But the key -point is you should know how Python 3 will treat every one one of your string -literals and you should mark them as appropriate. - -There are some differences between byte literals in Python 2 and those in -Python 3 thanks to the bytes type just being an alias to ``str`` in Python 2. -See the `Handle Common "Gotchas"`_ section for what to watch out for. - -``from __future__ import absolute_import`` -'''''''''''''''''''''''''''''''''''''''''' -Discussed in more detail below, but you should use this future statement to -prevent yourself from accidentally using implicit relative imports. - - -Supporting Python 2.5 and Newer Only -//////////////////////////////////// - -If you are supporting Python 2.5 and newer there are still some features of -Python that you can utilize. - - -``from __future__ import absolute_import`` -'''''''''''''''''''''''''''''''''''''''''' - -Implicit relative imports (e.g., importing ``spam.bacon`` from within -``spam.eggs`` with the statement ``import bacon``) do not work in Python 3. -This future statement moves away from that and allows the use of explicit -relative imports (e.g., ``from . import bacon``). - -In Python 2.5 you must use -the __future__ statement to get to use explicit relative imports and prevent -implicit ones. In Python 2.6 explicit relative imports are available without -the statement, but you still want the __future__ statement to prevent implicit -relative imports. In Python 2.7 the __future__ statement is not needed. In -other words, unless you are only supporting Python 2.7 or a version earlier -than Python 2.5, use this __future__ statement. - - -Mark all Unicode strings with a ``u`` prefix -''''''''''''''''''''''''''''''''''''''''''''' - -While Python 2.6 has a ``__future__`` statement to automatically cause Python 2 -to treat all string literals as Unicode, Python 2.5 does not have that shortcut. -This means you should go through and mark all string literals with a ``u`` -prefix to turn them explicitly into text strings where appropriate and only -support Python 3.3 or newer. Otherwise use a project like six_ which provides a -function to pass all text string literals through. - - -Capturing the Currently Raised Exception -'''''''''''''''''''''''''''''''''''''''' - -In Python 2.5 and earlier the syntax to access the current exception is:: - - try: - raise Exception() - except Exception, exc: - # Current exception is 'exc'. - pass - -This syntax changed in Python 3 (and backported to Python 2.6 and later) -to:: - - try: - raise Exception() - except Exception as exc: - # Current exception is 'exc'. - # In Python 3, 'exc' is restricted to the block; in Python 2.6/2.7 it will "leak". - pass - -Because of this syntax change you must change how you capture the current -exception in Python 2.5 and earlier to:: - - try: - raise Exception() - except Exception: - import sys - exc = sys.exc_info()[1] - # Current exception is 'exc'. - pass - -You can get more information about the raised exception from -:func:`sys.exc_info` than simply the current exception instance, but you most -likely don't need it. - -.. note:: - In Python 3, the traceback is attached to the exception instance - through the ``__traceback__`` attribute. If the instance is saved in - a local variable that persists outside of the ``except`` block, the - traceback will create a reference cycle with the current frame and its - dictionary of local variables. This will delay reclaiming dead - resources until the next cyclic :term:`garbage collection` pass. - - In Python 2, this problem only occurs if you save the traceback itself - (e.g. the third element of the tuple returned by :func:`sys.exc_info`) - in a variable. - - -Handle Common "Gotchas" -/////////////////////// - -These are things to watch out for no matter what version of Python 2 you are -supporting which are not syntactic considerations. - - -``from __future__ import division`` -''''''''''''''''''''''''''''''''''' - -While the exact same outcome can be had by using the ``-Qnew`` argument to -Python, using this future statement lifts the requirement that your users use -the flag to get the expected behavior of division in Python 3 -(e.g., ``1/2 == 0.5; 1//2 == 0``). - - - -Specify when opening a file as binary -''''''''''''''''''''''''''''''''''''' - +As part of this dichotomy you also need to be careful about opening files. Unless you have been working on Windows, there is a chance you have not always bothered to add the ``b`` mode when opening a binary file (e.g., ``rb`` for binary reading). Under Python 3, binary files and text files are clearly distinct and mutually incompatible; see the :mod:`io` module for details. Therefore, you **must** make a decision of whether a file will be used for -binary access (allowing to read and/or write bytes data) or text access -(allowing to read and/or write unicode data). +binary access (allowing to read and/or write binary data) or text access +(allowing to read and/or write text data). You should also use :func:`io.open` +for opening files instead of the built-in :func:`open` function as the :mod:`io` +module is consistent from Python 2 to 3 while the built-in :func:`open` function +is not (in Python 3 it's actually :func:`io.open`). -Text files -'''''''''' +Finally, the indexing of binary data requires careful handling (slicing does +**not** require any special handling). In Python 2, +``b'123'[1] == b'2'`` while in Python 3 ``b'123'[1] == 50``. Because binary data +is simply a collection of binary numbers, Python 3 returns the integer value for +the byte you index on. But in Python 2 because ``bytes == str``, indexing +returns a one-item slice of bytes. The six_ project has a function +named ``six.indexbytes()`` which will return an integer like in Python 3: +``six.indexbytes(b'123', 1)``. -Text files created using ``open()`` under Python 2 return byte strings, -while under Python 3 they return unicode strings. Depending on your porting -strategy, this can be an issue. +To summarize: -If you want text files to return unicode strings in Python 2, you have two -possibilities: +#. Decide which of your APIs take text and which take binary data +#. Make sure that your code that works with text also works with ``unicode`` and + code for binary data works with ``bytes`` in Python 2 (see the table above + for what methods you cannot use for each type) +#. Mark all binary literals with a ``b`` prefix, use a ``u`` prefix or + :mod:`__future__` import statement for text literals +#. Decode binary data to text as soon as possible, encode text as binary data as + late as possible +#. Open files using :func:`io.open` and make sure to specify the ``b`` mode when + appropriate +#. Be careful when indexing binary data -* Under Python 2.6 and higher, use :func:`io.open`. Since :func:`io.open` - is essentially the same function in both Python 2 and Python 3, it will - help iron out any issues that might arise. +Prevent compatibility regressions +--------------------------------- -* If pre-2.6 compatibility is needed, then you should use :func:`codecs.open` - instead. This will make sure that you get back unicode strings in Python 2. +Once you have fully translated your code to be compatible with Python 3, you +will want to make sure your code doesn't regress and stop working under +Python 3. This is especially true if you have a dependency which is blocking you +from actually running under Python 3 at the moment. -Subclass ``object`` -''''''''''''''''''' +To help with staying compatible, any new modules you create should have +at least the following block of code at the top of it:: -New-style classes have been around since Python 2.2. You need to make sure -you are subclassing from ``object`` to avoid odd edge cases involving method -resolution order, etc. This continues to be totally valid in Python 3 (although -unneeded as all classes implicitly inherit from ``object``). + from __future__ import absolute_import + from __future__ import division + from __future__ import print_statement + from __future__ import unicode_literals +You can also run Python 2 with the ``-3`` flag to be warned about various +compatibility issues your code triggers during execution. If you turn warnings +into errors with ``-Werror`` then you can make sure that you don't accidentally +miss a warning. -Deal With the Bytes/String Dichotomy -'''''''''''''''''''''''''''''''''''' -One of the biggest issues people have when porting code to Python 3 is handling -the bytes/string dichotomy. Because Python 2 allowed the ``str`` type to hold -textual data, people have over the years been rather loose in their delineation -of what ``str`` instances held text compared to bytes. In Python 3 you cannot -be so care-free anymore and need to properly handle the difference. The key to -handling this issue is to make sure that **every** string literal in your -Python 2 code is either syntactically or functionally marked as either bytes or -text data. After this is done you then need to make sure your APIs are designed -to either handle a specific type or made to be properly polymorphic. +You can also use the Pylint_ project and its ``--py3k`` flag to lint your code +to receive warnings when your code begins to deviate from Python 3 +compatibility. This also prevents you from having to run Modernize_ or Futurize_ +over your code regularly to catch compatibility regressions. This does require +you only support Python 2.7 and Python 3.4 or newer as that is Pylint's +minimum Python version support. -Mark Up Python 2 String Literals -******************************** +Check which dependencies block your transition +---------------------------------------------- -First thing you must do is designate every single string literal in Python 2 -as either textual or bytes data. If you are only supporting Python 2.6 or -newer, this can be accomplished by marking bytes literals with a ``b`` prefix -and then designating textual data with a ``u`` prefix or using the -``unicode_literals`` future statement. +**After** you have made your code compatible with Python 3 you should begin to +care about whether your dependencies have also been ported. The caniusepython3_ +project was created to help you determine which projects +-- directly or indirectly -- are blocking you from supporting Python 3. There +is both a command-line tool as well as a web interface at +https://caniusepython3.com . -If your project supports versions of Python predating 2.6, then you should use -the six_ project and its ``b()`` function to denote bytes literals. For text -literals you can either use six's ``u()`` function or use a ``u`` prefix. +The project also provides code which you can integrate into your test suite so +that you will have a failing test when you no longer have dependencies blocking +you from using Python 3. This allows you to avoid having to manually check your +dependencies and to be notified quickly when you can start running on Python 3. +Update your ``setup.py`` file to denote Python 3 compatibility +-------------------------------------------------------------- -Decide what APIs Will Accept -**************************** +Once your code works under Python 3, you should update the classifiers in +your ``setup.py`` to contain ``Programming Language :: Python :: 3`` and to not +specify sole Python 2 support. This will tell +anyone using your code that you support Python 2 **and** 3. Ideally you will +also want to add classifiers for each major/minor version of Python you now +support. -In Python 2 it was very easy to accidentally create an API that accepted both -bytes and textual data. But in Python 3, thanks to the more strict handling of -disparate types, this loose usage of bytes and text together tends to fail. +Use continuous integration to stay compatible +--------------------------------------------- -Take the dict ``{b'a': 'bytes', u'a': 'text'}`` in Python 2.6. It creates the -dict ``{u'a': 'text'}`` since ``b'a' == u'a'``. But in Python 3 the equivalent -dict creates ``{b'a': 'bytes', 'a': 'text'}``, i.e., no lost data. Similar -issues can crop up when transitioning Python 2 code to Python 3. +Once you are able to fully run under Python 3 you will want to make sure your +code always works under both Python 2 & 3. Probably the best tool for running +your tests under multiple Python interpreters is tox_. You can then integrate +tox with your continuous integration system so that you never accidentally break +Python 2 or 3 support. -This means you need to choose what an API is going to accept and create and -consistently stick to that API in both Python 2 and 3. +You may also want to use use the ``-bb`` flag with the Python 3 interpreter to +trigger an exception when you are comparing bytes to strings. Usually it's +simply ``False``, but if you made a mistake in your separation of text/binary +data handling you may be accidentally comparing text and binary data. This flag +will raise an exception when that occurs to help track down such cases. +And that's mostly it! At this point your code base is compatible with both +Python 2 and 3 simultaneously. Your testing will also be set up so that you +don't accidentally break Python 2 or 3 compatibility regardless of which version +you typically run your tests under while developing. -Bytes / Unicode Comparison -************************** -In Python 3, mixing bytes and unicode is forbidden in most situations; it -will raise a :class:`TypeError` where Python 2 would have attempted an implicit -coercion between types. However, there is one case where it doesn't and -it can be very misleading:: +Dropping Python 2 support completely +==================================== - >>> b"" == "" - False +If you are able to fully drop support for Python 2, then the steps required +to transition to Python 3 simplify greatly. -This is because an equality comparison is required by the language to always -succeed (and return ``False`` for incompatible types). However, this also -means that code incorrectly ported to Python 3 can display buggy behaviour -if such comparisons are silently executed. To detect such situations, -Python 3 has a ``-b`` flag that will display a warning:: +#. Update your code to only support Python 2.7 +#. Make sure you have good test coverage (coverage.py_ can help) +#. Learn the differences between Python 2 & 3 +#. Use 2to3_ to rewrite your code to run only under Python 3 - $ python3 -b - >>> b"" == "" - __main__:1: BytesWarning: Comparison between bytes and string - False +After this your code will be fully Python 3 compliant but in a way that is not +supported by Python 2. You should also update the classifiers in your +``setup.py`` to contain ``Programming Language :: Python :: 3 :: Only``. -To turn the warning into an exception, use the ``-bb`` flag instead:: - $ python3 -bb - >>> b"" == "" - Traceback (most recent call last): - File "", line 1, in - BytesWarning: Comparison between bytes and string +.. _2to3: https://docs.python.org/3/library/2to3.html +.. _caniusepython3: +.. _coverage.py: https://pypi.python.org/pypi/coverage +.. _Futurize: http://python-future.org/automatic_conversion.html +.. _Modernize: +.. _Porting to Python 3: http://python3porting.com/ +.. _Pylint: https://pypi.python.org/pypi/pylint +.. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html - -Indexing bytes objects -'''''''''''''''''''''' - -Another potentially surprising change is the indexing behaviour of bytes -objects in Python 3:: - - >>> b"xyz"[0] - 120 - -Indeed, Python 3 bytes objects (as well as :class:`bytearray` objects) -are sequences of integers. But code converted from Python 2 will often -assume that indexing a bytestring produces another bytestring, not an -integer. To reconcile both behaviours, use slicing:: - - >>> b"xyz"[0:1] - b'x' - >>> n = 1 - >>> b"xyz"[n:n+1] - b'y' - -The only remaining gotcha is that an out-of-bounds slice returns an empty -bytes object instead of raising ``IndexError``: - - >>> b"xyz"[3] - Traceback (most recent call last): - File "", line 1, in - IndexError: index out of range - >>> b"xyz"[3:4] - b'' - - -``__str__()``/``__unicode__()`` -''''''''''''''''''''''''''''''' - -In Python 2, objects can specify both a string and unicode representation of -themselves. In Python 3, though, there is only a string representation. This -becomes an issue as people can inadvertently do things in their ``__str__()`` -methods which have unpredictable results (e.g., infinite recursion if you -happen to use the ``unicode(self).encode('utf8')`` idiom as the body of your -``__str__()`` method). - -You can use a mixin class to work around this. This allows you to only define a -``__unicode__()`` method for your class and let the mixin derive -``__str__()`` for you (code from -http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/):: - - import sys - - class UnicodeMixin(object): - - """Mixin class to handle defining the proper __str__/__unicode__ - methods in Python 2 or 3.""" - - if sys.version_info[0] >= 3: # Python 3 - def __str__(self): - return self.__unicode__() - else: # Python 2 - def __str__(self): - return self.__unicode__().encode('utf8') - - - class Spam(UnicodeMixin): - - def __unicode__(self): - return u'spam-spam-bacon-spam' # 2to3 will remove the 'u' prefix - - -Don't Index on Exceptions -''''''''''''''''''''''''' - -In Python 2, the following worked:: - - >>> exc = Exception(1, 2, 3) - >>> exc.args[1] - 2 - >>> exc[1] # Python 2 only! - 2 - -But in Python 3, indexing directly on an exception is an error. You need to -make sure to only index on the :attr:`BaseException.args` attribute which is a -sequence containing all arguments passed to the :meth:`__init__` method. - -Even better is to use the documented attributes the exception provides. - - -Don't use ``__getslice__`` & Friends -'''''''''''''''''''''''''''''''''''' - -Been deprecated for a while, but Python 3 finally drops support for -``__getslice__()``, etc. Move completely over to :meth:`__getitem__` and -friends. - - -Updating doctests -''''''''''''''''' - -Don't forget to make them Python 2/3 compatible as well. If you wrote a -monolithic set of doctests (e.g., a single docstring containing all of your -doctests), you should at least consider breaking the doctests up into smaller -pieces to make it more manageable to fix. Otherwise it might very well be worth -your time and effort to port your tests to :mod:`unittest`. - - -Update ``map`` for imbalanced input sequences -''''''''''''''''''''''''''''''''''''''''''''' - -With Python 2, when ``map`` was given more than one input sequence it would pad -the shorter sequences with ``None`` values, returning a sequence as long as the -longest input sequence. - -With Python 3, if the input sequences to ``map`` are of unequal length, ``map`` -will stop at the termination of the shortest of the sequences. For full -compatibility with ``map`` from Python 2.x, wrap the sequence arguments in -:func:`itertools.zip_longest`, e.g. ``map(func, *sequences)`` becomes -``list(map(func, itertools.zip_longest(*sequences)))``. - -Eliminate ``-3`` Warnings -------------------------- - -When you run your application's test suite, run it using the ``-3`` flag passed -to Python. This will cause various warnings to be raised during execution about -things that are semantic changes between Python 2 and 3. Try to eliminate those -warnings to make your code even more portable to Python 3. - - -Alternative Approaches -====================== - -While supporting Python 2 & 3 simultaneously is typically the preferred choice -by people so that they can continue to improve code and have it work for the -most number of users, your life may be easier if you only have to support one -major version of Python going forward. - -Supporting Only Python 3 Going Forward From Python 2 Code ---------------------------------------------------------- - -If you have Python 2 code but going forward only want to improve it as Python 3 -code, then you can use :ref:`2to3 <2to3-reference>` to translate your Python 2 -code to Python 3 code. This is only recommended, though, if your current -version of your project is going into maintenance mode and you want all new -features to be exclusive to Python 3. - - -Backporting Python 3 code to Python 2 -------------------------------------- - -If you have Python 3 code and have little interest in supporting Python 2 you -can use 3to2_ to translate from Python 3 code to Python 2 code. This is only -recommended if you don't plan to heavily support Python 2 users. Otherwise -write your code for Python 3 and then backport as far back as you want. This -is typically easier than going from Python 2 to 3 as you will have worked out -any difficulties with e.g. bytes/strings, etc. - - -Other Resources -=============== - -The authors of the following blog posts, wiki pages, and books deserve special -thanks for making public their tips for porting Python 2 code to Python 3 (and -thus helping provide information for this document and its various revisions -over the years): - -* https://wiki.python.org/moin/PortingPythonToPy3k -* http://python3porting.com/ -* http://docs.pythonsprints.com/python3_porting/py-porting.html -* http://techspot.zzzeek.org/2011/01/24/zzzeek-s-guide-to-python-3-porting/ -* http://dabeaz.blogspot.com/2011/01/porting-py65-and-my-superboard-to.html -* http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ -* http://lucumr.pocoo.org/2010/2/11/porting-to-python-3-a-guide/ -* https://wiki.ubuntu.com/Python/3 - -If you feel there is something missing from this document that should be added, -please email the python-porting_ mailing list. - - -.. _3to2: https://pypi.python.org/pypi/3to2 -.. _Cheeseshop: PyPI_ -.. _coverage: https://pypi.python.org/pypi/coverage -.. _future: http://python-future.org/ -.. _modernize: https://github.com/mitsuhiko/python-modernize -.. _Porting to Python 3: http://python3porting.com/ -.. _PyPI: https://pypi.python.org/pypi -.. _Python 3 Packages: https://pypi.python.org/pypi?:action=browse&c=533&show=all -.. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html +.. _python-future: http://python-future.org/ .. _python-porting: https://mail.python.org/mailman/listinfo/python-porting .. _six: https://pypi.python.org/pypi/six .. _tox: https://pypi.python.org/pypi/tox -.. _trove classifiers: https://pypi.python.org/pypi?%3Aaction=list_classifiers +.. _trove classifier: https://pypi.python.org/pypi?%3Aaction=list_classifiers +.. _"What's New": https://docs.python.org/3/whatsnew/index.html diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -219,6 +219,9 @@ Documentation ------------- +- Issue #22914: Update the Python 2/3 porting HOWTO to describe a more automated + approach. + - Issue #21514: The documentation of the json module now refers to new JSON RFC 7159 instead of obsoleted RFC 4627. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 19:34:36 2014 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 05 Dec 2014 18:34:36 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Mark_PEP_479_as_accepted_and_?= =?utf-8?q?clarify_the_process=2E?= Message-ID: <20141205183432.74601.693@psf.io> https://hg.python.org/peps/rev/b130a6405af0 changeset: 5633:b130a6405af0 user: Guido van Rossum date: Fri Dec 05 10:34:06 2014 -0800 summary: Mark PEP 479 as accepted and clarify the process. files: pep-0479.txt | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-) diff --git a/pep-0479.txt b/pep-0479.txt --- a/pep-0479.txt +++ b/pep-0479.txt @@ -3,12 +3,12 @@ Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico , Guido van Rossum -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014 Python-Version: 3.5 -Post-History: 15-Nov-2014, 19-Nov-2014 +Post-History: 15-Nov-2014, 19-Nov-2014, 5-Dec-2014 Abstract @@ -22,6 +22,20 @@ ``__future__`` statement. +Acceptance +========== + +This PEP was accepted by the BDFL on November 22. Because of the +exceptionally short period from first draft to acceptance, the main +objections brought up after acceptance were carefully considered and +have been reflected in the "Alternate proposals" section below. +However, none of the discussion changed the BDFL's mind and the PEP's +acceptance is now final. (Suggestions for clarifying edits are still +welcome -- unlike IETF RFCs, the text of a PEP is not cast in stone +after its acceptance, although the core design/plan/specification +should not change after acceptance.) + + Rationale ========= -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Fri Dec 5 21:17:45 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 05 Dec 2014 20:17:45 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322394=3A_Add_a_?= =?utf-8?q?=27venv=27_command_to_Doc/Makefile=2E?= Message-ID: <20141205201742.73384.24853@psf.io> https://hg.python.org/cpython/rev/888103600e72 changeset: 93739:888103600e72 user: Brett Cannon date: Fri Dec 05 15:17:31 2014 -0500 summary: Issue #22394: Add a 'venv' command to Doc/Makefile. This will create a venv using the interpreter specified by the PYTHON variable for the Makefile that also install Sphinx. Typical usage is expected to be: cd Doc make venv PYTHON=../python make html PYTHON=venv/bin/python3 files: .gitignore | 6 +----- .hgignore | 1 + Doc/Makefile | 10 +++++++--- Misc/NEWS | 4 ++++ 4 files changed, 13 insertions(+), 8 deletions(-) diff --git a/.gitignore b/.gitignore --- a/.gitignore +++ b/.gitignore @@ -9,11 +9,7 @@ *~ .gdb_history Doc/build/ -Doc/tools/docutils/ -Doc/tools/jinja/ -Doc/tools/jinja2/ -Doc/tools/pygments/ -Doc/tools/sphinx/ +Doc/venv/ Lib/lib2to3/*.pickle Lib/test/data/* Lib/_sysconfigdata.py diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -9,6 +9,7 @@ autom4te.cache$ ^build/ ^Doc/build/ +^Doc/venv/ buildno$ config.cache config.log diff --git a/Doc/Makefile b/Doc/Makefile --- a/Doc/Makefile +++ b/Doc/Makefile @@ -15,11 +15,12 @@ .PHONY: help build html htmlhelp latex text changes linkcheck \ suspicious coverage doctest pydoc-topics htmlview clean dist check serve \ - autobuild-dev autobuild-stable + autobuild-dev autobuild-stable venv help: @echo "Please use \`make ' where is one of" @echo " clean to remove build files" + @echo " venv to create a venv with necessary tools" @echo " html to make standalone HTML files" @echo " htmlview to open the index page built by the html target in your browser" @echo " htmlhelp to make HTML files and a HTML help project" @@ -102,7 +103,11 @@ $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" clean: - -rm -rf build/* + -rm -rf build/* venv/* + +venv: + $(PYTHON) -m venv venv + ./venv/bin/python3 -m pip install -U Sphinx dist: rm -rf dist @@ -172,4 +177,3 @@ exit 1;; \ esac @make autobuild-dev - diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1348,6 +1348,10 @@ Documentation ------------- +- Issue #22394: Doc/Makefile now supports ``make venv PYTHON=../python`` to + create a venv for generating the documentation, e.g., + ``make html PYTHON=venv/bin/python3``. + - Issue #21514: The documentation of the json module now refers to new JSON RFC 7159 instead of obsoleted RFC 4627. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 21:30:38 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 05 Dec 2014 20:30:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2322581=3A_Use_more_=22bytes-like_object=22_throu?= =?utf-8?q?ghout_the_docs_and_comments=2E?= Message-ID: <20141205203038.76724.36421@psf.io> https://hg.python.org/cpython/rev/450a025b1669 changeset: 93741:450a025b1669 parent: 93738:021c1df36910 parent: 93740:853e3e115db4 user: Serhiy Storchaka date: Fri Dec 05 22:26:10 2014 +0200 summary: Issue #22581: Use more "bytes-like object" throughout the docs and comments. files: Doc/c-api/arg.rst | 27 +++++++++--------- Doc/c-api/unicode.rst | 3 +- Doc/library/socket.rst | 5 ++- Doc/library/ssl.rst | 2 +- Include/unicodeobject.h | 2 +- Lib/multiprocessing/connection.py | 2 +- Modules/arraymodule.c | 4 +- Modules/socketmodule.c | 6 ++-- Objects/stringlib/join.h | 6 ++-- Objects/unicodeobject.c | 3 +- Python/getargs.c | 14 ++++---- Python/marshal.c | 2 +- 12 files changed, 39 insertions(+), 37 deletions(-) diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -65,19 +65,20 @@ :exc:`UnicodeError` is raised. .. note:: - This format does not accept bytes-like objects. If you want to accept + This format does not accept :term:`bytes-like objects + `. If you want to accept filesystem paths and convert them to C character strings, it is preferable to use the ``O&`` format with :c:func:`PyUnicode_FSConverter` as *converter*. -``s*`` (:class:`str`, :class:`bytes`, :class:`bytearray` or buffer compatible object) [Py_buffer] - This format accepts Unicode objects as well as :term:`bytes-like object`\ s. +``s*`` (:class:`str` or :term:`bytes-like object`) [Py_buffer] + This format accepts Unicode objects as well as bytes-like objects. It fills a :c:type:`Py_buffer` structure provided by the caller. In this case the resulting C string may contain embedded NUL bytes. Unicode objects are converted to C strings using ``'utf-8'`` encoding. -``s#`` (:class:`str`, :class:`bytes` or read-only buffer compatible object) [const char \*, int or :c:type:`Py_ssize_t`] - Like ``s*``, except that it doesn't accept mutable buffer-like objects +``s#`` (:class:`str`, read-only :term:`bytes-like object`) [const char \*, int or :c:type:`Py_ssize_t`] + Like ``s*``, except that it doesn't accept mutable bytes-like objects such as :class:`bytearray`. The result is stored into two C variables, the first one a pointer to a C string, the second one its length. The string may contain embedded null bytes. Unicode objects are converted @@ -87,28 +88,28 @@ Like ``s``, but the Python object may also be ``None``, in which case the C pointer is set to *NULL*. -``z*`` (:class:`str`, :class:`bytes`, :class:`bytearray`, buffer compatible object or ``None``) [Py_buffer] +``z*`` (:class:`str`, :term:`bytes-like object` or ``None``) [Py_buffer] Like ``s*``, but the Python object may also be ``None``, in which case the ``buf`` member of the :c:type:`Py_buffer` structure is set to *NULL*. -``z#`` (:class:`str`, :class:`bytes`, read-only buffer compatible object or ``None``) [const char \*, int] +``z#`` (:class:`str`, read-only :term:`bytes-like object` or ``None``) [const char \*, int] Like ``s#``, but the Python object may also be ``None``, in which case the C pointer is set to *NULL*. -``y`` (:class:`bytes`) [const char \*] +``y`` (read-only :term:`bytes-like object`) [const char \*] This format converts a bytes-like object to a C pointer to a character string; it does not accept Unicode objects. The bytes buffer must not contain embedded NUL bytes; if it does, a :exc:`TypeError` exception is raised. -``y*`` (:class:`bytes`, :class:`bytearray` or :term:`bytes-like object`) [Py_buffer] +``y*`` (:term:`bytes-like object`) [Py_buffer] This variant on ``s*`` doesn't accept Unicode objects, only - :term:`bytes-like object`\ s. **This is the recommended way to accept + bytes-like objects. **This is the recommended way to accept binary data.** -``y#`` (:class:`bytes`) [const char \*, int] - This variant on ``s#`` doesn't accept Unicode objects, only :term:`bytes-like - object`\ s. +``y#`` (read-only :term:`bytes-like object`) [const char \*, int] + This variant on ``s#`` doesn't accept Unicode objects, only bytes-like + objects. ``S`` (:class:`bytes`) [PyBytesObject \*] Requires that the Python object is a :class:`bytes` object, without diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -556,7 +556,8 @@ Coerce an encoded object *obj* to an Unicode object and return a reference with incremented refcount. - :class:`bytes`, :class:`bytearray` and other char buffer compatible objects + :class:`bytes`, :class:`bytearray` and other + :term:`bytes-like objects ` are decoded according to the given *encoding* and using the error handling defined by *errors*. Both can be *NULL* to have the interface use the default values (see the next section for details). diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -1123,7 +1123,8 @@ Send normal and ancillary data to the socket, gathering the non-ancillary data from a series of buffers and concatenating it into a single message. The *buffers* argument specifies the - non-ancillary data as an iterable of buffer-compatible objects + non-ancillary data as an iterable of + :term:`bytes-like objects ` (e.g. :class:`bytes` objects); the operating system may set a limit (:func:`~os.sysconf` value ``SC_IOV_MAX``) on the number of buffers that can be used. The *ancdata* argument specifies the ancillary @@ -1131,7 +1132,7 @@ ``(cmsg_level, cmsg_type, cmsg_data)``, where *cmsg_level* and *cmsg_type* are integers specifying the protocol level and protocol-specific type respectively, and *cmsg_data* is a - buffer-compatible object holding the associated data. Note that + bytes-like object holding the associated data. Note that some systems (in particular, systems without :func:`CMSG_SPACE`) might support sending only one control message per call. The *flags* argument defaults to 0 and has the same meaning as for diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1104,7 +1104,7 @@ `_. The *cadata* object, if present, is either an ASCII string of one or more - PEM-encoded certificates or a bytes-like object of DER-encoded + PEM-encoded certificates or a :term:`bytes-like object` of DER-encoded certificates. Like with *capath* extra lines around PEM-encoded certificates are ignored but at least one certificate must be present. diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -849,7 +849,7 @@ Coercion is done in the following way: - 1. bytes, bytearray and other char buffer compatible objects are decoded + 1. bytes, bytearray and other bytes-like objects are decoded under the assumptions that they contain data using the UTF-8 encoding. Decoding is done in "strict" mode. diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -220,7 +220,7 @@ def recv_bytes_into(self, buf, offset=0): """ - Receive bytes data into a writeable buffer-like object. + Receive bytes data into a writeable bytes-like object. Return the number of bytes read. """ self._check_closed() diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -1573,14 +1573,14 @@ Py_ssize_t n; if (buffer->itemsize != 1) { PyBuffer_Release(buffer); - PyErr_SetString(PyExc_TypeError, "string/buffer of bytes required."); + PyErr_SetString(PyExc_TypeError, "a bytes-like object is required"); return NULL; } n = buffer->len; if (n % itemsize != 0) { PyBuffer_Release(buffer); PyErr_SetString(PyExc_ValueError, - "string length not a multiple of item size"); + "bytes length not a multiple of item size"); return NULL; } n = n / itemsize; diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -3560,7 +3560,7 @@ for (; ndatabufs < ndataparts; ndatabufs++) { if (!PyArg_Parse(PySequence_Fast_GET_ITEM(data_fast, ndatabufs), "y*;sendmsg() argument 1 must be an iterable of " - "buffer-compatible objects", + "bytes-like objects", &databufs[ndatabufs])) goto finally; iovs[ndatabufs].iov_base = databufs[ndatabufs].buf; @@ -3717,12 +3717,12 @@ Send normal and ancillary data to the socket, gathering the\n\ non-ancillary data from a series of buffers and concatenating it into\n\ a single message. The buffers argument specifies the non-ancillary\n\ -data as an iterable of buffer-compatible objects (e.g. bytes objects).\n\ +data as an iterable of bytes-like objects (e.g. bytes objects).\n\ The ancdata argument specifies the ancillary data (control messages)\n\ as an iterable of zero or more tuples (cmsg_level, cmsg_type,\n\ cmsg_data), where cmsg_level and cmsg_type are integers specifying the\n\ protocol level and protocol-specific type respectively, and cmsg_data\n\ -is a buffer-compatible object holding the associated data. The flags\n\ +is a bytes-like object holding the associated data. The flags\n\ argument defaults to 0 and has the same meaning as for send(). If\n\ address is supplied and not None, it sets a destination address for\n\ the message. The return value is the number of bytes of non-ancillary\n\ diff --git a/Objects/stringlib/join.h b/Objects/stringlib/join.h --- a/Objects/stringlib/join.h +++ b/Objects/stringlib/join.h @@ -53,15 +53,15 @@ /* Here is the general case. Do a pre-pass to figure out the total * amount of space we'll need (sz), and see whether all arguments are - * buffer-compatible. + * bytes-like. */ for (i = 0, nbufs = 0; i < seqlen; i++) { Py_ssize_t itemlen; item = PySequence_Fast_GET_ITEM(seq, i); if (_getbuffer(item, &buffers[i]) < 0) { PyErr_Format(PyExc_TypeError, - "sequence item %zd: expected bytes, bytearray, " - "or an object with the buffer interface, %.80s found", + "sequence item %zd: expected a bytes-like object, " + "%.80s found", i, Py_TYPE(item)->tp_name); goto error; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -2905,8 +2905,7 @@ /* Retrieve a bytes buffer view through the PEP 3118 buffer interface */ if (PyObject_GetBuffer(obj, &buffer, PyBUF_SIMPLE) < 0) { PyErr_Format(PyExc_TypeError, - "coercing to str: need bytes, bytearray " - "or buffer-like object, %.80s found", + "coercing to str: need a bytes-like object, %.80s found", Py_TYPE(obj)->tp_name); return NULL; } diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -849,7 +849,7 @@ /* XXX WAAAAH! 's', 'y', 'z', 'u', 'Z', 'e', 'w' codes all need to be cleaned up! */ - case 'y': {/* any buffer-like object, but not PyUnicode */ + case 'y': {/* any bytes-like object */ void **p = (void **)va_arg(*p_va, char **); char *buf; Py_ssize_t count; @@ -880,8 +880,8 @@ break; } - case 's': /* text string */ - case 'z': /* text string or None */ + case 's': /* text string or bytes-like object */ + case 'z': /* text string, bytes-like object or None */ { if (*format == '*') { /* "s*" or "z*" */ @@ -897,7 +897,7 @@ arg, msgbuf, bufsize); PyBuffer_FillInfo(p, arg, sarg, len, 1, 0); } - else { /* any buffer-like object */ + else { /* any bytes-like object */ char *buf; if (getbuffer(arg, p, &buf) < 0) return converterr(buf, arg, msgbuf, bufsize); @@ -908,7 +908,7 @@ arg, msgbuf, bufsize); } format++; - } else if (*format == '#') { /* any buffer-like object */ + } else if (*format == '#') { /* a string or read-only bytes-like object */ /* "s#" or "z#" */ void **p = (void **)va_arg(*p_va, char **); FETCH_SIZE; @@ -926,7 +926,7 @@ *p = sarg; STORE_SIZE(len); } - else { /* any buffer-like object */ + else { /* read-only bytes-like object */ /* XXX Really? */ char *buf; Py_ssize_t count = convertbuffer(arg, p, &buf); @@ -966,7 +966,7 @@ { Py_UNICODE **p = va_arg(*p_va, Py_UNICODE **); - if (*format == '#') { /* any buffer-like object */ + if (*format == '#') { /* "s#" or "Z#" */ FETCH_SIZE; diff --git a/Python/marshal.c b/Python/marshal.c --- a/Python/marshal.c +++ b/Python/marshal.c @@ -527,7 +527,7 @@ w_object(co->co_lnotab, p); } else if (PyObject_CheckBuffer(v)) { - /* Write unknown buffer-style objects as a string */ + /* Write unknown bytes-like objects as a byte string */ Py_buffer view; if (PyObject_GetBuffer(v, &view, PyBUF_SIMPLE) != 0) { w_byte(TYPE_UNKNOWN, p); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 21:30:38 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 05 Dec 2014 20:30:38 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyNTgx?= =?utf-8?q?=3A_Use_more_=22bytes-like_object=22_throughout_the_docs_and_co?= =?utf-8?q?mments=2E?= Message-ID: <20141205203037.27519.99983@psf.io> https://hg.python.org/cpython/rev/853e3e115db4 changeset: 93740:853e3e115db4 branch: 3.4 parent: 93737:55b94462ca7f user: Serhiy Storchaka date: Fri Dec 05 22:25:22 2014 +0200 summary: Issue #22581: Use more "bytes-like object" throughout the docs and comments. files: Doc/c-api/arg.rst | 27 +++++++++--------- Doc/c-api/unicode.rst | 3 +- Doc/library/socket.rst | 5 ++- Doc/library/ssl.rst | 2 +- Include/unicodeobject.h | 2 +- Lib/multiprocessing/connection.py | 2 +- Modules/arraymodule.c | 4 +- Modules/socketmodule.c | 6 ++-- Objects/stringlib/join.h | 6 ++-- Objects/unicodeobject.c | 3 +- Python/getargs.c | 14 ++++---- Python/marshal.c | 2 +- 12 files changed, 39 insertions(+), 37 deletions(-) diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -65,19 +65,20 @@ :exc:`UnicodeError` is raised. .. note:: - This format does not accept bytes-like objects. If you want to accept + This format does not accept :term:`bytes-like objects + `. If you want to accept filesystem paths and convert them to C character strings, it is preferable to use the ``O&`` format with :c:func:`PyUnicode_FSConverter` as *converter*. -``s*`` (:class:`str`, :class:`bytes`, :class:`bytearray` or buffer compatible object) [Py_buffer] - This format accepts Unicode objects as well as :term:`bytes-like object`\ s. +``s*`` (:class:`str` or :term:`bytes-like object`) [Py_buffer] + This format accepts Unicode objects as well as bytes-like objects. It fills a :c:type:`Py_buffer` structure provided by the caller. In this case the resulting C string may contain embedded NUL bytes. Unicode objects are converted to C strings using ``'utf-8'`` encoding. -``s#`` (:class:`str`, :class:`bytes` or read-only buffer compatible object) [const char \*, int or :c:type:`Py_ssize_t`] - Like ``s*``, except that it doesn't accept mutable buffer-like objects +``s#`` (:class:`str`, read-only :term:`bytes-like object`) [const char \*, int or :c:type:`Py_ssize_t`] + Like ``s*``, except that it doesn't accept mutable bytes-like objects such as :class:`bytearray`. The result is stored into two C variables, the first one a pointer to a C string, the second one its length. The string may contain embedded null bytes. Unicode objects are converted @@ -87,28 +88,28 @@ Like ``s``, but the Python object may also be ``None``, in which case the C pointer is set to *NULL*. -``z*`` (:class:`str`, :class:`bytes`, :class:`bytearray`, buffer compatible object or ``None``) [Py_buffer] +``z*`` (:class:`str`, :term:`bytes-like object` or ``None``) [Py_buffer] Like ``s*``, but the Python object may also be ``None``, in which case the ``buf`` member of the :c:type:`Py_buffer` structure is set to *NULL*. -``z#`` (:class:`str`, :class:`bytes`, read-only buffer compatible object or ``None``) [const char \*, int] +``z#`` (:class:`str`, read-only :term:`bytes-like object` or ``None``) [const char \*, int] Like ``s#``, but the Python object may also be ``None``, in which case the C pointer is set to *NULL*. -``y`` (:class:`bytes`) [const char \*] +``y`` (read-only :term:`bytes-like object`) [const char \*] This format converts a bytes-like object to a C pointer to a character string; it does not accept Unicode objects. The bytes buffer must not contain embedded NUL bytes; if it does, a :exc:`TypeError` exception is raised. -``y*`` (:class:`bytes`, :class:`bytearray` or :term:`bytes-like object`) [Py_buffer] +``y*`` (:term:`bytes-like object`) [Py_buffer] This variant on ``s*`` doesn't accept Unicode objects, only - :term:`bytes-like object`\ s. **This is the recommended way to accept + bytes-like objects. **This is the recommended way to accept binary data.** -``y#`` (:class:`bytes`) [const char \*, int] - This variant on ``s#`` doesn't accept Unicode objects, only :term:`bytes-like - object`\ s. +``y#`` (read-only :term:`bytes-like object`) [const char \*, int] + This variant on ``s#`` doesn't accept Unicode objects, only bytes-like + objects. ``S`` (:class:`bytes`) [PyBytesObject \*] Requires that the Python object is a :class:`bytes` object, without diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -556,7 +556,8 @@ Coerce an encoded object *obj* to an Unicode object and return a reference with incremented refcount. - :class:`bytes`, :class:`bytearray` and other char buffer compatible objects + :class:`bytes`, :class:`bytearray` and other + :term:`bytes-like objects ` are decoded according to the given *encoding* and using the error handling defined by *errors*. Both can be *NULL* to have the interface use the default values (see the next section for details). diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -1118,7 +1118,8 @@ Send normal and ancillary data to the socket, gathering the non-ancillary data from a series of buffers and concatenating it into a single message. The *buffers* argument specifies the - non-ancillary data as an iterable of buffer-compatible objects + non-ancillary data as an iterable of + :term:`bytes-like objects ` (e.g. :class:`bytes` objects); the operating system may set a limit (:func:`~os.sysconf` value ``SC_IOV_MAX``) on the number of buffers that can be used. The *ancdata* argument specifies the ancillary @@ -1126,7 +1127,7 @@ ``(cmsg_level, cmsg_type, cmsg_data)``, where *cmsg_level* and *cmsg_type* are integers specifying the protocol level and protocol-specific type respectively, and *cmsg_data* is a - buffer-compatible object holding the associated data. Note that + bytes-like object holding the associated data. Note that some systems (in particular, systems without :func:`CMSG_SPACE`) might support sending only one control message per call. The *flags* argument defaults to 0 and has the same meaning as for diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1072,7 +1072,7 @@ `_. The *cadata* object, if present, is either an ASCII string of one or more - PEM-encoded certificates or a bytes-like object of DER-encoded + PEM-encoded certificates or a :term:`bytes-like object` of DER-encoded certificates. Like with *capath* extra lines around PEM-encoded certificates are ignored but at least one certificate must be present. diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -849,7 +849,7 @@ Coercion is done in the following way: - 1. bytes, bytearray and other char buffer compatible objects are decoded + 1. bytes, bytearray and other bytes-like objects are decoded under the assumptions that they contain data using the UTF-8 encoding. Decoding is done in "strict" mode. diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -220,7 +220,7 @@ def recv_bytes_into(self, buf, offset=0): """ - Receive bytes data into a writeable buffer-like object. + Receive bytes data into a writeable bytes-like object. Return the number of bytes read. """ self._check_closed() diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -1441,14 +1441,14 @@ Py_ssize_t n; if (buffer->itemsize != 1) { PyBuffer_Release(buffer); - PyErr_SetString(PyExc_TypeError, "string/buffer of bytes required."); + PyErr_SetString(PyExc_TypeError, "a bytes-like object is required"); return NULL; } n = buffer->len; if (n % itemsize != 0) { PyBuffer_Release(buffer); PyErr_SetString(PyExc_ValueError, - "string length not a multiple of item size"); + "bytes length not a multiple of item size"); return NULL; } n = n / itemsize; diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -3493,7 +3493,7 @@ for (; ndatabufs < ndataparts; ndatabufs++) { if (!PyArg_Parse(PySequence_Fast_GET_ITEM(data_fast, ndatabufs), "y*;sendmsg() argument 1 must be an iterable of " - "buffer-compatible objects", + "bytes-like objects", &databufs[ndatabufs])) goto finally; iovs[ndatabufs].iov_base = databufs[ndatabufs].buf; @@ -3650,12 +3650,12 @@ Send normal and ancillary data to the socket, gathering the\n\ non-ancillary data from a series of buffers and concatenating it into\n\ a single message. The buffers argument specifies the non-ancillary\n\ -data as an iterable of buffer-compatible objects (e.g. bytes objects).\n\ +data as an iterable of bytes-like objects (e.g. bytes objects).\n\ The ancdata argument specifies the ancillary data (control messages)\n\ as an iterable of zero or more tuples (cmsg_level, cmsg_type,\n\ cmsg_data), where cmsg_level and cmsg_type are integers specifying the\n\ protocol level and protocol-specific type respectively, and cmsg_data\n\ -is a buffer-compatible object holding the associated data. The flags\n\ +is a bytes-like object holding the associated data. The flags\n\ argument defaults to 0 and has the same meaning as for send(). If\n\ address is supplied and not None, it sets a destination address for\n\ the message. The return value is the number of bytes of non-ancillary\n\ diff --git a/Objects/stringlib/join.h b/Objects/stringlib/join.h --- a/Objects/stringlib/join.h +++ b/Objects/stringlib/join.h @@ -53,15 +53,15 @@ /* Here is the general case. Do a pre-pass to figure out the total * amount of space we'll need (sz), and see whether all arguments are - * buffer-compatible. + * bytes-like. */ for (i = 0, nbufs = 0; i < seqlen; i++) { Py_ssize_t itemlen; item = PySequence_Fast_GET_ITEM(seq, i); if (_getbuffer(item, &buffers[i]) < 0) { PyErr_Format(PyExc_TypeError, - "sequence item %zd: expected bytes, bytearray, " - "or an object with the buffer interface, %.80s found", + "sequence item %zd: expected a bytes-like object, " + "%.80s found", i, Py_TYPE(item)->tp_name); goto error; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -2940,8 +2940,7 @@ /* Retrieve a bytes buffer view through the PEP 3118 buffer interface */ if (PyObject_GetBuffer(obj, &buffer, PyBUF_SIMPLE) < 0) { PyErr_Format(PyExc_TypeError, - "coercing to str: need bytes, bytearray " - "or buffer-like object, %.80s found", + "coercing to str: need a bytes-like object, %.80s found", Py_TYPE(obj)->tp_name); return NULL; } diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -849,7 +849,7 @@ /* XXX WAAAAH! 's', 'y', 'z', 'u', 'Z', 'e', 'w' codes all need to be cleaned up! */ - case 'y': {/* any buffer-like object, but not PyUnicode */ + case 'y': {/* any bytes-like object */ void **p = (void **)va_arg(*p_va, char **); char *buf; Py_ssize_t count; @@ -880,8 +880,8 @@ break; } - case 's': /* text string */ - case 'z': /* text string or None */ + case 's': /* text string or bytes-like object */ + case 'z': /* text string, bytes-like object or None */ { if (*format == '*') { /* "s*" or "z*" */ @@ -897,7 +897,7 @@ arg, msgbuf, bufsize); PyBuffer_FillInfo(p, arg, sarg, len, 1, 0); } - else { /* any buffer-like object */ + else { /* any bytes-like object */ char *buf; if (getbuffer(arg, p, &buf) < 0) return converterr(buf, arg, msgbuf, bufsize); @@ -908,7 +908,7 @@ arg, msgbuf, bufsize); } format++; - } else if (*format == '#') { /* any buffer-like object */ + } else if (*format == '#') { /* a string or read-only bytes-like object */ /* "s#" or "z#" */ void **p = (void **)va_arg(*p_va, char **); FETCH_SIZE; @@ -926,7 +926,7 @@ *p = sarg; STORE_SIZE(len); } - else { /* any buffer-like object */ + else { /* read-only bytes-like object */ /* XXX Really? */ char *buf; Py_ssize_t count = convertbuffer(arg, p, &buf); @@ -967,7 +967,7 @@ { Py_UNICODE **p = va_arg(*p_va, Py_UNICODE **); - if (*format == '#') { /* any buffer-like object */ + if (*format == '#') { /* "s#" or "Z#" */ FETCH_SIZE; diff --git a/Python/marshal.c b/Python/marshal.c --- a/Python/marshal.c +++ b/Python/marshal.c @@ -529,7 +529,7 @@ w_object(co->co_lnotab, p); } else if (PyObject_CheckBuffer(v)) { - /* Write unknown buffer-style objects as a string */ + /* Write unknown bytes-like objects as a byte string */ Py_buffer view; if (PyObject_GetBuffer(v, &view, PyBUF_SIMPLE) != 0) { w_byte(TYPE_UNKNOWN, p); -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 21:30:38 2014 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 05 Dec 2014 20:30:38 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <20141205203038.76708.37265@psf.io> https://hg.python.org/cpython/rev/6a4d07282a24 changeset: 93742:6a4d07282a24 parent: 93741:450a025b1669 parent: 93739:888103600e72 user: Serhiy Storchaka date: Fri Dec 05 22:28:27 2014 +0200 summary: Merge heads files: .gitignore | 6 +----- .hgignore | 1 + Doc/Makefile | 10 +++++++--- Misc/NEWS | 4 ++++ 4 files changed, 13 insertions(+), 8 deletions(-) diff --git a/.gitignore b/.gitignore --- a/.gitignore +++ b/.gitignore @@ -9,11 +9,7 @@ *~ .gdb_history Doc/build/ -Doc/tools/docutils/ -Doc/tools/jinja/ -Doc/tools/jinja2/ -Doc/tools/pygments/ -Doc/tools/sphinx/ +Doc/venv/ Lib/lib2to3/*.pickle Lib/test/data/* Lib/_sysconfigdata.py diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -9,6 +9,7 @@ autom4te.cache$ ^build/ ^Doc/build/ +^Doc/venv/ buildno$ config.cache config.log diff --git a/Doc/Makefile b/Doc/Makefile --- a/Doc/Makefile +++ b/Doc/Makefile @@ -15,11 +15,12 @@ .PHONY: help build html htmlhelp latex text changes linkcheck \ suspicious coverage doctest pydoc-topics htmlview clean dist check serve \ - autobuild-dev autobuild-stable + autobuild-dev autobuild-stable venv help: @echo "Please use \`make ' where is one of" @echo " clean to remove build files" + @echo " venv to create a venv with necessary tools" @echo " html to make standalone HTML files" @echo " htmlview to open the index page built by the html target in your browser" @echo " htmlhelp to make HTML files and a HTML help project" @@ -102,7 +103,11 @@ $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" clean: - -rm -rf build/* + -rm -rf build/* venv/* + +venv: + $(PYTHON) -m venv venv + ./venv/bin/python3 -m pip install -U Sphinx dist: rm -rf dist @@ -172,4 +177,3 @@ exit 1;; \ esac @make autobuild-dev - diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1348,6 +1348,10 @@ Documentation ------------- +- Issue #22394: Doc/Makefile now supports ``make venv PYTHON=../python`` to + create a venv for generating the documentation, e.g., + ``make html PYTHON=venv/bin/python3``. + - Issue #21514: The documentation of the json module now refers to new JSON RFC 7159 instead of obsoleted RFC 4627. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Fri Dec 5 22:52:32 2014 From: python-checkins at python.org (victor.stinner) Date: Fri, 05 Dec 2014 21:52:32 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=239647=3A_os=2Econf?= =?utf-8?q?str=28=29_ensures_that_the_second_call_to_confstr=28=29_returns?= =?utf-8?q?_the?= Message-ID: <20141205215223.21252.98891@psf.io> https://hg.python.org/cpython/rev/a7a8947e9ce4 changeset: 93743:a7a8947e9ce4 user: Victor Stinner date: Fri Dec 05 22:51:51 2014 +0100 summary: Issue #9647: os.confstr() ensures that the second call to confstr() returns the same length. files: Modules/posixmodule.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -14369,10 +14369,12 @@ } if (len >= sizeof(buffer)) { + size_t len2; char *buf = PyMem_Malloc(len); if (buf == NULL) return PyErr_NoMemory(); - confstr(name, buf, len); + len2 = confstr(name, buf, len); + assert(len == len2); result = PyUnicode_DecodeFSDefaultAndSize(buf, len-1); PyMem_Free(buf); } -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 00:11:31 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 05 Dec 2014 23:11:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_a_bad_link?= Message-ID: <20141205231128.73394.84380@psf.io> https://hg.python.org/cpython/rev/db6c9a5329f2 changeset: 93744:db6c9a5329f2 branch: 3.4 parent: 93740:853e3e115db4 user: Brett Cannon date: Fri Dec 05 18:11:05 2014 -0500 summary: Fix a bad link files: Doc/howto/pyporting.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -361,7 +361,7 @@ .. _2to3: https://docs.python.org/3/library/2to3.html -.. _caniusepython3: +.. _caniusepython3: https://pypi.python.org/pypi/caniusepython3 .. _coverage.py: https://pypi.python.org/pypi/coverage .. _Futurize: http://python-future.org/automatic_conversion.html .. _Modernize: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 00:11:31 2014 From: python-checkins at python.org (brett.cannon) Date: Fri, 05 Dec 2014 23:11:31 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E4?= Message-ID: <20141205231128.27517.48700@psf.io> https://hg.python.org/cpython/rev/fc7dd575fd94 changeset: 93745:fc7dd575fd94 parent: 93743:a7a8947e9ce4 parent: 93744:db6c9a5329f2 user: Brett Cannon date: Fri Dec 05 18:11:15 2014 -0500 summary: Merge with 3.4 files: Doc/howto/pyporting.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -361,7 +361,7 @@ .. _2to3: https://docs.python.org/3/library/2to3.html -.. _caniusepython3: +.. _caniusepython3: https://pypi.python.org/pypi/caniusepython3 .. _coverage.py: https://pypi.python.org/pypi/coverage .. _Futurize: http://python-future.org/automatic_conversion.html .. _Modernize: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:17:00 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 01:17:00 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogaW4gcG9wbGliLCBs?= =?utf-8?q?imit_maximum_line_length_that_we_read_from_the_network_=28close?= =?utf-8?q?s?= Message-ID: <20141206011659.76718.94653@psf.io> https://hg.python.org/cpython/rev/339f877cca11 changeset: 93746:339f877cca11 branch: 2.7 parent: 93707:54af09408795 user: Benjamin Peterson date: Fri Dec 05 20:02:38 2014 -0500 summary: in poplib, limit maximum line length that we read from the network (closes #16041) Patch from Berker Peksag. files: Lib/poplib.py | 12 +++++++++++- Lib/test/test_poplib.py | 4 ++++ Misc/NEWS | 3 +++ 3 files changed, 18 insertions(+), 1 deletions(-) diff --git a/Lib/poplib.py b/Lib/poplib.py --- a/Lib/poplib.py +++ b/Lib/poplib.py @@ -32,6 +32,12 @@ LF = '\n' CRLF = CR+LF +# maximal line length when calling readline(). This is to prevent +# reading arbitrary length lines. RFC 1939 limits POP3 line length to +# 512 characters, including CRLF. We have selected 2048 just to be on +# the safe side. +_MAXLINE = 2048 + class POP3: @@ -103,7 +109,9 @@ # Raise error_proto('-ERR EOF') if the connection is closed. def _getline(self): - line = self.file.readline() + line = self.file.readline(_MAXLINE + 1) + if len(line) > _MAXLINE: + raise error_proto('line too long') if self._debugging > 1: print '*get*', repr(line) if not line: raise error_proto('-ERR EOF') octets = len(line) @@ -365,6 +373,8 @@ match = renewline.match(self.buffer) while not match: self._fillBuffer() + if len(self.buffer) > _MAXLINE: + raise error_proto('line too long') match = renewline.match(self.buffer) line = match.group(0) self.buffer = renewline.sub('' ,self.buffer, 1) diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -198,6 +198,10 @@ 113) self.assertEqual(self.client.retr('foo'), expected) + def test_too_long_lines(self): + self.assertRaises(poplib.error_proto, self.client._shortcmd, + 'echo +%s' % ((poplib._MAXLINE + 10) * 'a')) + def test_dele(self): self.assertOK(self.client.dele('foo')) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Library ------- +- Issue #16041: In poplib, limit maximum line length read from the server to + prevent CVE-2013-1752. + - Issue #22960: Add a context argument to xmlrpclib.ServerProxy. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:16:59 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 01:16:59 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogc210cGxpYjogbGlt?= =?utf-8?q?it_amount_read_from_the_network_=28closes_=2316042=29?= Message-ID: <20141206011659.21254.42960@psf.io> https://hg.python.org/cpython/rev/923aac88a3cc changeset: 93747:923aac88a3cc branch: 2.7 user: Benjamin Peterson date: Fri Dec 05 20:05:18 2014 -0500 summary: smtplib: limit amount read from the network (closes #16042) files: Lib/smtplib.py | 11 ++++++++- Lib/test/test_smtplib.py | 30 +++++++++++++++++++++++++++- Misc/NEWS | 3 ++ 3 files changed, 41 insertions(+), 3 deletions(-) diff --git a/Lib/smtplib.py b/Lib/smtplib.py --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -57,6 +57,7 @@ SMTP_PORT = 25 SMTP_SSL_PORT = 465 CRLF = "\r\n" +_MAXLINE = 8192 # more than 8 times larger than RFC 821, 4.5.3 OLDSTYLE_AUTH = re.compile(r"auth=(.*)", re.I) @@ -179,10 +180,14 @@ def __init__(self, sslobj): self.sslobj = sslobj - def readline(self): + def readline(self, size=-1): + if size < 0: + size = None str = "" chr = None while chr != "\n": + if size is not None and len(str) >= size: + break chr = self.sslobj.read(1) if not chr: break @@ -353,7 +358,7 @@ self.file = self.sock.makefile('rb') while 1: try: - line = self.file.readline() + line = self.file.readline(_MAXLINE + 1) except socket.error as e: self.close() raise SMTPServerDisconnected("Connection unexpectedly closed: " @@ -363,6 +368,8 @@ raise SMTPServerDisconnected("Connection unexpectedly closed") if self.debuglevel > 0: print>>stderr, 'reply:', repr(line) + if len(line) > _MAXLINE: + raise SMTPResponseException(500, "Line too long.") resp.append(line[4:].strip()) code = line[:3] # Check that the error code is syntactically correct. diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -292,6 +292,33 @@ HOST, self.port, 'localhost', 3) + at unittest.skipUnless(threading, 'Threading required for this test.') +class TooLongLineTests(unittest.TestCase): + respdata = '250 OK' + ('.' * smtplib._MAXLINE * 2) + '\n' + + def setUp(self): + self.old_stdout = sys.stdout + self.output = StringIO.StringIO() + sys.stdout = self.output + + self.evt = threading.Event() + self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.sock.settimeout(15) + self.port = test_support.bind_port(self.sock) + servargs = (self.evt, self.respdata, self.sock) + threading.Thread(target=server, args=servargs).start() + self.evt.wait() + self.evt.clear() + + def tearDown(self): + self.evt.wait() + sys.stdout = self.old_stdout + + def testLineTooLong(self): + self.assertRaises(smtplib.SMTPResponseException, smtplib.SMTP, + HOST, self.port, 'localhost', 3) + + sim_users = {'Mr.A at somewhere.com':'John A', 'Ms.B at somewhere.com':'Sally B', 'Mrs.C at somewhereesle.com':'Ruth C', @@ -526,7 +553,8 @@ def test_main(verbose=None): test_support.run_unittest(GeneralTests, DebuggingServerTests, NonConnectingTests, - BadHELOServerTests, SMTPSimTests) + BadHELOServerTests, SMTPSimTests, + TooLongLineTests) if __name__ == '__main__': test_main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Library ------- +- Issue #16042: CVE-2013-1752: smtplib: Limit amount of data read by limiting + the call to readline(). Original patch by Christian Heimes. + - Issue #16041: In poplib, limit maximum line length read from the server to prevent CVE-2013-1752. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:17:00 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 01:17:00 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_add_a_default_?= =?utf-8?q?limit_for_the_amount_of_data_xmlrpclib=2Egzip=5Fdecode_will_ret?= =?utf-8?q?urn?= Message-ID: <20141206011659.21260.64043@psf.io> https://hg.python.org/cpython/rev/d50096708b2d changeset: 93748:d50096708b2d branch: 2.7 user: Benjamin Peterson date: Fri Dec 05 20:15:15 2014 -0500 summary: add a default limit for the amount of data xmlrpclib.gzip_decode will return (closes #16043) files: Lib/test/test_xmlrpc.py | 19 ++++++++++++++++++- Lib/xmlrpclib.py | 13 +++++++++++-- Misc/NEWS | 3 +++ 3 files changed, 32 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -737,7 +737,7 @@ with cm: p.pow(6, 8) - def test_gsip_response(self): + def test_gzip_response(self): t = self.Transport() p = xmlrpclib.ServerProxy(URL, transport=t) old = self.requestHandler.encode_threshold @@ -750,6 +750,23 @@ self.requestHandler.encode_threshold = old self.assertTrue(a>b) + def test_gzip_decode_limit(self): + max_gzip_decode = 20 * 1024 * 1024 + data = '\0' * max_gzip_decode + encoded = xmlrpclib.gzip_encode(data) + decoded = xmlrpclib.gzip_decode(encoded) + self.assertEqual(len(decoded), max_gzip_decode) + + data = '\0' * (max_gzip_decode + 1) + encoded = xmlrpclib.gzip_encode(data) + + with self.assertRaisesRegexp(ValueError, + "max gzipped payload length exceeded"): + xmlrpclib.gzip_decode(encoded) + + xmlrpclib.gzip_decode(encoded, max_decode=-1) + + #Test special attributes of the ServerProxy object class ServerProxyTestCase(unittest.TestCase): def setUp(self): diff --git a/Lib/xmlrpclib.py b/Lib/xmlrpclib.py --- a/Lib/xmlrpclib.py +++ b/Lib/xmlrpclib.py @@ -49,6 +49,7 @@ # 2003-07-12 gp Correct marshalling of Faults # 2003-10-31 mvl Add multicall support # 2004-08-20 mvl Bump minimum supported Python version to 2.1 +# 2014-12-02 ch/doko Add workaround for gzip bomb vulnerability # # Copyright (c) 1999-2002 by Secret Labs AB. # Copyright (c) 1999-2002 by Fredrik Lundh. @@ -1165,10 +1166,13 @@ # in the HTTP header, as described in RFC 1952 # # @param data The encoded data +# @keyparam max_decode Maximum bytes to decode (20MB default), use negative +# values for unlimited decoding # @return the unencoded data # @raises ValueError if data is not correctly coded. +# @raises ValueError if max gzipped payload length exceeded -def gzip_decode(data): +def gzip_decode(data, max_decode=20971520): """gzip encoded data -> unencoded data Decode data using the gzip content encoding as described in RFC 1952 @@ -1178,11 +1182,16 @@ f = StringIO.StringIO(data) gzf = gzip.GzipFile(mode="rb", fileobj=f) try: - decoded = gzf.read() + if max_decode < 0: # no limit + decoded = gzf.read() + else: + decoded = gzf.read(max_decode + 1) except IOError: raise ValueError("invalid data") f.close() gzf.close() + if max_decode >= 0 and len(decoded) > max_decode: + raise ValueError("max gzipped payload length exceeded") return decoded ## diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Library ------- +- Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode + will return. This resolves CVE-2013-1753. + - Issue #16042: CVE-2013-1752: smtplib: Limit amount of data read by limiting the call to readline(). Original patch by Christian Heimes. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:17:01 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 01:17:01 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_merge_2=2E7=2E9_release_branch?= Message-ID: <20141206011700.74599.55672@psf.io> https://hg.python.org/cpython/rev/88af4cb9d187 changeset: 93749:88af4cb9d187 branch: 2.7 parent: 93732:84928af5d703 parent: 93748:d50096708b2d user: Benjamin Peterson date: Fri Dec 05 20:16:53 2014 -0500 summary: merge 2.7.9 release branch files: Lib/poplib.py | 12 ++++++++++- Lib/smtplib.py | 11 ++++++++- Lib/test/test_poplib.py | 4 +++ Lib/test/test_smtplib.py | 30 +++++++++++++++++++++++++++- Lib/test/test_xmlrpc.py | 19 ++++++++++++++++- Lib/xmlrpclib.py | 13 ++++++++++- Misc/NEWS | 9 ++++++++ 7 files changed, 91 insertions(+), 7 deletions(-) diff --git a/Lib/poplib.py b/Lib/poplib.py --- a/Lib/poplib.py +++ b/Lib/poplib.py @@ -32,6 +32,12 @@ LF = '\n' CRLF = CR+LF +# maximal line length when calling readline(). This is to prevent +# reading arbitrary length lines. RFC 1939 limits POP3 line length to +# 512 characters, including CRLF. We have selected 2048 just to be on +# the safe side. +_MAXLINE = 2048 + class POP3: @@ -103,7 +109,9 @@ # Raise error_proto('-ERR EOF') if the connection is closed. def _getline(self): - line = self.file.readline() + line = self.file.readline(_MAXLINE + 1) + if len(line) > _MAXLINE: + raise error_proto('line too long') if self._debugging > 1: print '*get*', repr(line) if not line: raise error_proto('-ERR EOF') octets = len(line) @@ -365,6 +373,8 @@ match = renewline.match(self.buffer) while not match: self._fillBuffer() + if len(self.buffer) > _MAXLINE: + raise error_proto('line too long') match = renewline.match(self.buffer) line = match.group(0) self.buffer = renewline.sub('' ,self.buffer, 1) diff --git a/Lib/smtplib.py b/Lib/smtplib.py --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -57,6 +57,7 @@ SMTP_PORT = 25 SMTP_SSL_PORT = 465 CRLF = "\r\n" +_MAXLINE = 8192 # more than 8 times larger than RFC 821, 4.5.3 OLDSTYLE_AUTH = re.compile(r"auth=(.*)", re.I) @@ -179,10 +180,14 @@ def __init__(self, sslobj): self.sslobj = sslobj - def readline(self): + def readline(self, size=-1): + if size < 0: + size = None str = "" chr = None while chr != "\n": + if size is not None and len(str) >= size: + break chr = self.sslobj.read(1) if not chr: break @@ -353,7 +358,7 @@ self.file = self.sock.makefile('rb') while 1: try: - line = self.file.readline() + line = self.file.readline(_MAXLINE + 1) except socket.error as e: self.close() raise SMTPServerDisconnected("Connection unexpectedly closed: " @@ -363,6 +368,8 @@ raise SMTPServerDisconnected("Connection unexpectedly closed") if self.debuglevel > 0: print>>stderr, 'reply:', repr(line) + if len(line) > _MAXLINE: + raise SMTPResponseException(500, "Line too long.") resp.append(line[4:].strip()) code = line[:3] # Check that the error code is syntactically correct. diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -198,6 +198,10 @@ 113) self.assertEqual(self.client.retr('foo'), expected) + def test_too_long_lines(self): + self.assertRaises(poplib.error_proto, self.client._shortcmd, + 'echo +%s' % ((poplib._MAXLINE + 10) * 'a')) + def test_dele(self): self.assertOK(self.client.dele('foo')) diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -292,6 +292,33 @@ HOST, self.port, 'localhost', 3) + at unittest.skipUnless(threading, 'Threading required for this test.') +class TooLongLineTests(unittest.TestCase): + respdata = '250 OK' + ('.' * smtplib._MAXLINE * 2) + '\n' + + def setUp(self): + self.old_stdout = sys.stdout + self.output = StringIO.StringIO() + sys.stdout = self.output + + self.evt = threading.Event() + self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.sock.settimeout(15) + self.port = test_support.bind_port(self.sock) + servargs = (self.evt, self.respdata, self.sock) + threading.Thread(target=server, args=servargs).start() + self.evt.wait() + self.evt.clear() + + def tearDown(self): + self.evt.wait() + sys.stdout = self.old_stdout + + def testLineTooLong(self): + self.assertRaises(smtplib.SMTPResponseException, smtplib.SMTP, + HOST, self.port, 'localhost', 3) + + sim_users = {'Mr.A at somewhere.com':'John A', 'Ms.B at somewhere.com':'Sally B', 'Mrs.C at somewhereesle.com':'Ruth C', @@ -526,7 +553,8 @@ def test_main(verbose=None): test_support.run_unittest(GeneralTests, DebuggingServerTests, NonConnectingTests, - BadHELOServerTests, SMTPSimTests) + BadHELOServerTests, SMTPSimTests, + TooLongLineTests) if __name__ == '__main__': test_main() diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -737,7 +737,7 @@ with cm: p.pow(6, 8) - def test_gsip_response(self): + def test_gzip_response(self): t = self.Transport() p = xmlrpclib.ServerProxy(URL, transport=t) old = self.requestHandler.encode_threshold @@ -750,6 +750,23 @@ self.requestHandler.encode_threshold = old self.assertTrue(a>b) + def test_gzip_decode_limit(self): + max_gzip_decode = 20 * 1024 * 1024 + data = '\0' * max_gzip_decode + encoded = xmlrpclib.gzip_encode(data) + decoded = xmlrpclib.gzip_decode(encoded) + self.assertEqual(len(decoded), max_gzip_decode) + + data = '\0' * (max_gzip_decode + 1) + encoded = xmlrpclib.gzip_encode(data) + + with self.assertRaisesRegexp(ValueError, + "max gzipped payload length exceeded"): + xmlrpclib.gzip_decode(encoded) + + xmlrpclib.gzip_decode(encoded, max_decode=-1) + + #Test special attributes of the ServerProxy object class ServerProxyTestCase(unittest.TestCase): def setUp(self): diff --git a/Lib/xmlrpclib.py b/Lib/xmlrpclib.py --- a/Lib/xmlrpclib.py +++ b/Lib/xmlrpclib.py @@ -49,6 +49,7 @@ # 2003-07-12 gp Correct marshalling of Faults # 2003-10-31 mvl Add multicall support # 2004-08-20 mvl Bump minimum supported Python version to 2.1 +# 2014-12-02 ch/doko Add workaround for gzip bomb vulnerability # # Copyright (c) 1999-2002 by Secret Labs AB. # Copyright (c) 1999-2002 by Fredrik Lundh. @@ -1165,10 +1166,13 @@ # in the HTTP header, as described in RFC 1952 # # @param data The encoded data +# @keyparam max_decode Maximum bytes to decode (20MB default), use negative +# values for unlimited decoding # @return the unencoded data # @raises ValueError if data is not correctly coded. +# @raises ValueError if max gzipped payload length exceeded -def gzip_decode(data): +def gzip_decode(data, max_decode=20971520): """gzip encoded data -> unencoded data Decode data using the gzip content encoding as described in RFC 1952 @@ -1178,11 +1182,16 @@ f = StringIO.StringIO(data) gzf = gzip.GzipFile(mode="rb", fileobj=f) try: - decoded = gzf.read() + if max_decode < 0: # no limit + decoded = gzf.read() + else: + decoded = gzf.read(max_decode + 1) except IOError: raise ValueError("invalid data") f.close() gzf.close() + if max_decode >= 0 and len(decoded) > max_decode: + raise ValueError("max gzipped payload length exceeded") return decoded ## diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -52,6 +52,15 @@ Library ------- +- Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode + will return. This resolves CVE-2013-1753. + +- Issue #16042: CVE-2013-1752: smtplib: Limit amount of data read by limiting + the call to readline(). Original patch by Christian Heimes. + +- Issue #16041: In poplib, limit maximum line length read from the server to + prevent CVE-2013-1752. + - Issue #22960: Add a context argument to xmlrpclib.ServerProxy. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:26:48 2014 From: python-checkins at python.org (terry.reedy) Date: Sat, 06 Dec 2014 01:26:48 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE2ODkz?= =?utf-8?q?=3A_Update_Idle_doc_chapter_to_match_current_Idle_and_add_new?= Message-ID: <20141206012642.73396.564@psf.io> https://hg.python.org/cpython/rev/2fc341c6e314 changeset: 93751:2fc341c6e314 branch: 3.4 parent: 93744:db6c9a5329f2 user: Terry Jan Reedy date: Fri Dec 05 20:25:30 2014 -0500 summary: Issue #16893: Update Idle doc chapter to match current Idle and add new information. files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -175,6 +175,9 @@ IDLE ---- +- Issue #16893: Update Idle doc chapter to match current Idle and add new + information. + - Issue #3068: Add Idle extension configuration dialog to Options menu. Changes are written to HOME/.idlerc/config-extensions.cfg. Original patch by Tal Einat. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:26:48 2014 From: python-checkins at python.org (terry.reedy) Date: Sat, 06 Dec 2014 01:26:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogTWVyZ2Ugd2l0aCAzLjQu?= Message-ID: <20141206012642.21276.42756@psf.io> https://hg.python.org/cpython/rev/a21c5b2020ce changeset: 93752:a21c5b2020ce parent: 93745:fc7dd575fd94 parent: 93751:2fc341c6e314 user: Terry Jan Reedy date: Fri Dec 05 20:26:23 2014 -0500 summary: Merge with 3.4. files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1195,6 +1195,9 @@ IDLE ---- +- Issue #16893: Update Idle doc chapter to match current Idle and add new + information. + - Issue #3068: Add Idle extension configuration dialog to Options menu. Changes are written to HOME/.idlerc/config-extensions.cfg. Original patch by Tal Einat. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:26:48 2014 From: python-checkins at python.org (terry.reedy) Date: Sat, 06 Dec 2014 01:26:48 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2ODkz?= =?utf-8?q?=3A_Update_Idle_doc_chapter_to_match_current_Idle_and_add_new?= Message-ID: <20141206012642.74597.76258@psf.io> https://hg.python.org/cpython/rev/283c364c372a changeset: 93750:283c364c372a branch: 2.7 user: Terry Jan Reedy date: Fri Dec 05 20:25:14 2014 -0500 summary: Issue #16893: Update Idle doc chapter to match current Idle and add new information. files: Misc/NEWS | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,12 @@ - Issue #22960: Add a context argument to xmlrpclib.ServerProxy. +IDLE +---- + +- Issue #16893: Update Idle doc chapter to match current Idle and add new + information. + What's New in Python 2.7.9 release candidate 1? =============================================== -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:36:14 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 01:36:14 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?= =?utf-8?q?_merge_3=2E3_=28=2316043=29?= Message-ID: <20141206013613.21254.1413@psf.io> https://hg.python.org/cpython/rev/6b83e21c8679 changeset: 93755:6b83e21c8679 branch: 3.4 parent: 93751:2fc341c6e314 parent: 93754:4a9418c6f8ae user: Benjamin Peterson date: Fri Dec 05 20:34:56 2014 -0500 summary: merge 3.3 (#16043) files: Lib/test/test_xmlrpc.py | 25 +++++++++++++++++++++++-- Lib/xmlrpc/client.py | 13 +++++++++++-- Misc/NEWS | 3 +++ 3 files changed, 37 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -863,7 +863,7 @@ p.pow(6, 8) p("close")() - def test_gsip_response(self): + def test_gzip_response(self): t = self.Transport() p = xmlrpclib.ServerProxy(URL, transport=t) old = self.requestHandler.encode_threshold @@ -877,6 +877,27 @@ self.requestHandler.encode_threshold = old self.assertTrue(a>b) + + at unittest.skipIf(gzip is None, 'requires gzip') +class GzipUtilTestCase(unittest.TestCase): + + def test_gzip_decode_limit(self): + max_gzip_decode = 20 * 1024 * 1024 + data = b'\0' * max_gzip_decode + encoded = xmlrpclib.gzip_encode(data) + decoded = xmlrpclib.gzip_decode(encoded) + self.assertEqual(len(decoded), max_gzip_decode) + + data = b'\0' * (max_gzip_decode + 1) + encoded = xmlrpclib.gzip_encode(data) + + with self.assertRaisesRegexp(ValueError, + "max gzipped payload length exceeded"): + xmlrpclib.gzip_decode(encoded) + + xmlrpclib.gzip_decode(encoded, max_decode=-1) + + #Test special attributes of the ServerProxy object class ServerProxyTestCase(unittest.TestCase): def setUp(self): @@ -1105,7 +1126,7 @@ support.run_unittest(XMLRPCTestCase, HelperTestCase, DateTimeTestCase, BinaryTestCase, FaultTestCase, UseBuiltinTypesTestCase, SimpleServerTestCase, KeepaliveServerTestCase1, - KeepaliveServerTestCase2, GzipServerTestCase, + KeepaliveServerTestCase2, GzipServerTestCase, GzipUtilTestCase, MultiPathServerTestCase, ServerProxyTestCase, FailingServerTestCase, CGIHandlerTestCase) diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -49,6 +49,7 @@ # 2003-07-12 gp Correct marshalling of Faults # 2003-10-31 mvl Add multicall support # 2004-08-20 mvl Bump minimum supported Python version to 2.1 +# 2014-12-02 ch/doko Add workaround for gzip bomb vulnerability # # Copyright (c) 1999-2002 by Secret Labs AB. # Copyright (c) 1999-2002 by Fredrik Lundh. @@ -1030,10 +1031,13 @@ # in the HTTP header, as described in RFC 1952 # # @param data The encoded data +# @keyparam max_decode Maximum bytes to decode (20MB default), use negative +# values for unlimited decoding # @return the unencoded data # @raises ValueError if data is not correctly coded. +# @raises ValueError if max gzipped payload length exceeded -def gzip_decode(data): +def gzip_decode(data, max_decode=20971520): """gzip encoded data -> unencoded data Decode data using the gzip content encoding as described in RFC 1952 @@ -1043,11 +1047,16 @@ f = BytesIO(data) gzf = gzip.GzipFile(mode="rb", fileobj=f) try: - decoded = gzf.read() + if max_decode < 0: # no limit + decoded = gzf.read() + else: + decoded = gzf.read(max_decode + 1) except OSError: raise ValueError("invalid data") f.close() gzf.close() + if max_decode >= 0 and len(decoded) > max_decode: + raise ValueError("max gzipped payload length exceeded") return decoded ## diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -39,6 +39,9 @@ Library ------- +- Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode + will return. This resolves CVE-2013-1753. + - Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects returned by ZipFile.open() can now operate independently of the ZipFile even if the ZipFile was created by passing in a file-like object as the first -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:36:15 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 01:36:15 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMTYwNDMp?= Message-ID: <20141206013613.76722.90287@psf.io> https://hg.python.org/cpython/rev/6f002c4741e2 changeset: 93756:6f002c4741e2 parent: 93752:a21c5b2020ce parent: 93755:6b83e21c8679 user: Benjamin Peterson date: Fri Dec 05 20:36:07 2014 -0500 summary: merge 3.4 (#16043) files: Lib/test/test_xmlrpc.py | 25 +++++++++++++++++++++++-- Lib/xmlrpc/client.py | 13 +++++++++++-- Misc/NEWS | 3 +++ 3 files changed, 37 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -880,7 +880,7 @@ p.pow(6, 8) p("close")() - def test_gsip_response(self): + def test_gzip_response(self): t = self.Transport() p = xmlrpclib.ServerProxy(URL, transport=t) old = self.requestHandler.encode_threshold @@ -894,6 +894,27 @@ self.requestHandler.encode_threshold = old self.assertTrue(a>b) + + at unittest.skipIf(gzip is None, 'requires gzip') +class GzipUtilTestCase(unittest.TestCase): + + def test_gzip_decode_limit(self): + max_gzip_decode = 20 * 1024 * 1024 + data = b'\0' * max_gzip_decode + encoded = xmlrpclib.gzip_encode(data) + decoded = xmlrpclib.gzip_decode(encoded) + self.assertEqual(len(decoded), max_gzip_decode) + + data = b'\0' * (max_gzip_decode + 1) + encoded = xmlrpclib.gzip_encode(data) + + with self.assertRaisesRegexp(ValueError, + "max gzipped payload length exceeded"): + xmlrpclib.gzip_decode(encoded) + + xmlrpclib.gzip_decode(encoded, max_decode=-1) + + #Test special attributes of the ServerProxy object class ServerProxyTestCase(unittest.TestCase): def setUp(self): @@ -1123,7 +1144,7 @@ support.run_unittest(XMLRPCTestCase, HelperTestCase, DateTimeTestCase, BinaryTestCase, FaultTestCase, UseBuiltinTypesTestCase, SimpleServerTestCase, KeepaliveServerTestCase1, - KeepaliveServerTestCase2, GzipServerTestCase, + KeepaliveServerTestCase2, GzipServerTestCase, GzipUtilTestCase, MultiPathServerTestCase, ServerProxyTestCase, FailingServerTestCase, CGIHandlerTestCase) diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -49,6 +49,7 @@ # 2003-07-12 gp Correct marshalling of Faults # 2003-10-31 mvl Add multicall support # 2004-08-20 mvl Bump minimum supported Python version to 2.1 +# 2014-12-02 ch/doko Add workaround for gzip bomb vulnerability # # Copyright (c) 1999-2002 by Secret Labs AB. # Copyright (c) 1999-2002 by Fredrik Lundh. @@ -1031,10 +1032,13 @@ # in the HTTP header, as described in RFC 1952 # # @param data The encoded data +# @keyparam max_decode Maximum bytes to decode (20MB default), use negative +# values for unlimited decoding # @return the unencoded data # @raises ValueError if data is not correctly coded. +# @raises ValueError if max gzipped payload length exceeded -def gzip_decode(data): +def gzip_decode(data, max_decode=20971520): """gzip encoded data -> unencoded data Decode data using the gzip content encoding as described in RFC 1952 @@ -1044,11 +1048,16 @@ f = BytesIO(data) gzf = gzip.GzipFile(mode="rb", fileobj=f) try: - decoded = gzf.read() + if max_decode < 0: # no limit + decoded = gzf.read() + else: + decoded = gzf.read(max_decode + 1) except OSError: raise ValueError("invalid data") f.close() gzf.close() + if max_decode >= 0 and len(decoded) > max_decode: + raise ValueError("max gzipped payload length exceeded") return decoded ## diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -194,6 +194,9 @@ Library ------- +- Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode + will return. This resolves CVE-2013-1753. + - Issue #14099: ZipFile.open() no longer reopen the underlying file. Objects returned by ZipFile.open() can now operate independently of the ZipFile even if the ZipFile was created by passing in a file-like object as the first -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:36:15 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 01:36:15 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_3=2E2_=28=2316043=29?= Message-ID: <20141206013612.27505.51455@psf.io> https://hg.python.org/cpython/rev/4a9418c6f8ae changeset: 93754:4a9418c6f8ae branch: 3.3 parent: 93393:6c46859edfd7 parent: 93753:a0368f81af9a user: Benjamin Peterson date: Fri Dec 05 20:30:54 2014 -0500 summary: merge 3.2 (#16043) files: Lib/test/test_xmlrpc.py | 23 ++++++++++++++++++++++- Lib/xmlrpc/client.py | 13 +++++++++++-- Misc/NEWS | 3 +++ 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -845,7 +845,7 @@ p.pow(6, 8) p("close")() - def test_gsip_response(self): + def test_gzip_response(self): t = self.Transport() p = xmlrpclib.ServerProxy(URL, transport=t) old = self.requestHandler.encode_threshold @@ -859,6 +859,26 @@ self.requestHandler.encode_threshold = old self.assertTrue(a>b) + +class GzipUtilTestCase(unittest.TestCase): + + def test_gzip_decode_limit(self): + max_gzip_decode = 20 * 1024 * 1024 + data = b'\0' * max_gzip_decode + encoded = xmlrpclib.gzip_encode(data) + decoded = xmlrpclib.gzip_decode(encoded) + self.assertEqual(len(decoded), max_gzip_decode) + + data = b'\0' * (max_gzip_decode + 1) + encoded = xmlrpclib.gzip_encode(data) + + with self.assertRaisesRegexp(ValueError, + "max gzipped payload length exceeded"): + xmlrpclib.gzip_decode(encoded) + + xmlrpclib.gzip_decode(encoded, max_decode=-1) + + #Test special attributes of the ServerProxy object class ServerProxyTestCase(unittest.TestCase): def setUp(self): @@ -1093,6 +1113,7 @@ try: import gzip xmlrpc_tests.append(GzipServerTestCase) + xmlrpc_tests.append(GzipUtilTestCase) except ImportError: pass #gzip not supported in this build xmlrpc_tests.append(MultiPathServerTestCase) diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -49,6 +49,7 @@ # 2003-07-12 gp Correct marshalling of Faults # 2003-10-31 mvl Add multicall support # 2004-08-20 mvl Bump minimum supported Python version to 2.1 +# 2014-12-02 ch/doko Add workaround for gzip bomb vulnerability # # Copyright (c) 1999-2002 by Secret Labs AB. # Copyright (c) 1999-2002 by Fredrik Lundh. @@ -1031,10 +1032,13 @@ # in the HTTP header, as described in RFC 1952 # # @param data The encoded data +# @keyparam max_decode Maximum bytes to decode (20MB default), use negative +# values for unlimited decoding # @return the unencoded data # @raises ValueError if data is not correctly coded. +# @raises ValueError if max gzipped payload length exceeded -def gzip_decode(data): +def gzip_decode(data, max_decode=20971520): """gzip encoded data -> unencoded data Decode data using the gzip content encoding as described in RFC 1952 @@ -1044,11 +1048,16 @@ f = BytesIO(data) gzf = gzip.GzipFile(mode="rb", fileobj=f) try: - decoded = gzf.read() + if max_decode < 0: # no limit + decoded = gzf.read() + else: + decoded = gzf.read(max_decode + 1) except IOError: raise ValueError("invalid data") f.close() gzf.close() + if max_decode >= 0 and len(decoded) > max_decode: + raise ValueError("max gzipped payload length exceeded") return decoded ## diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,9 @@ Library ------- +- Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode + will return. This resolves CVE-2013-1753. + - Issue #22517: When a io.BufferedRWPair object is deallocated, clear its weakrefs. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:36:14 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 01:36:14 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_add_a_default_?= =?utf-8?q?limit_for_the_amount_of_data_xmlrpclib=2Egzip=5Fdecode_will_ret?= =?utf-8?q?urn?= Message-ID: <20141206013612.76726.22287@psf.io> https://hg.python.org/cpython/rev/a0368f81af9a changeset: 93753:a0368f81af9a branch: 3.2 parent: 93392:4fbf1a7c1e40 user: Benjamin Peterson date: Fri Dec 05 20:15:15 2014 -0500 summary: add a default limit for the amount of data xmlrpclib.gzip_decode will return (closes #16043) files: Lib/test/test_xmlrpc.py | 23 ++++++++++++++++++++++- Lib/xmlrpc/client.py | 13 +++++++++++-- Misc/NEWS | 3 +++ 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -776,7 +776,7 @@ p.pow(6, 8) p("close")() - def test_gsip_response(self): + def test_gzip_response(self): t = self.Transport() p = xmlrpclib.ServerProxy(URL, transport=t) old = self.requestHandler.encode_threshold @@ -790,6 +790,26 @@ self.requestHandler.encode_threshold = old self.assertTrue(a>b) + +class GzipUtilTestCase(unittest.TestCase): + + def test_gzip_decode_limit(self): + max_gzip_decode = 20 * 1024 * 1024 + data = b'\0' * max_gzip_decode + encoded = xmlrpclib.gzip_encode(data) + decoded = xmlrpclib.gzip_decode(encoded) + self.assertEqual(len(decoded), max_gzip_decode) + + data = b'\0' * (max_gzip_decode + 1) + encoded = xmlrpclib.gzip_encode(data) + + with self.assertRaisesRegexp(ValueError, + "max gzipped payload length exceeded"): + xmlrpclib.gzip_decode(encoded) + + xmlrpclib.gzip_decode(encoded, max_decode=-1) + + #Test special attributes of the ServerProxy object class ServerProxyTestCase(unittest.TestCase): def setUp(self): @@ -990,6 +1010,7 @@ try: import gzip xmlrpc_tests.append(GzipServerTestCase) + xmlrpc_tests.append(GzipUtilTestCase) except ImportError: pass #gzip not supported in this build xmlrpc_tests.append(MultiPathServerTestCase) diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -49,6 +49,7 @@ # 2003-07-12 gp Correct marshalling of Faults # 2003-10-31 mvl Add multicall support # 2004-08-20 mvl Bump minimum supported Python version to 2.1 +# 2014-12-02 ch/doko Add workaround for gzip bomb vulnerability # # Copyright (c) 1999-2002 by Secret Labs AB. # Copyright (c) 1999-2002 by Fredrik Lundh. @@ -1017,10 +1018,13 @@ # in the HTTP header, as described in RFC 1952 # # @param data The encoded data +# @keyparam max_decode Maximum bytes to decode (20MB default), use negative +# values for unlimited decoding # @return the unencoded data # @raises ValueError if data is not correctly coded. +# @raises ValueError if max gzipped payload length exceeded -def gzip_decode(data): +def gzip_decode(data, max_decode=20971520): """gzip encoded data -> unencoded data Decode data using the gzip content encoding as described in RFC 1952 @@ -1030,11 +1034,16 @@ f = BytesIO(data) gzf = gzip.GzipFile(mode="rb", fileobj=f) try: - decoded = gzf.read() + if max_decode < 0: # no limit + decoded = gzf.read() + else: + decoded = gzf.read(max_decode + 1) except IOError: raise ValueError("invalid data") f.close() gzf.close() + if max_decode >= 0 and len(decoded) > max_decode: + raise ValueError("max gzipped payload length exceeded") return decoded ## diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -16,6 +16,9 @@ Library ------- +- Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode + will return. This resolves CVE-2013-1753. + - Issue #16040: CVE-2013-1752: nntplib: Limit maximum line lengths to 2048 to prevent readline() calls from consuming too much memory. Patch by Jyrki Pulliainen. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:51:27 2014 From: python-checkins at python.org (terry.reedy) Date: Sat, 06 Dec 2014 01:51:27 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Update_idlelib?= =?utf-8?q?/NEWS=2Etxt=2E?= Message-ID: <20141206015127.27503.36229@psf.io> https://hg.python.org/cpython/rev/14ab927a10f6 changeset: 93757:14ab927a10f6 branch: 2.7 parent: 93750:283c364c372a user: Terry Jan Reedy date: Fri Dec 05 20:49:23 2014 -0500 summary: Update idlelib/NEWS.txt. files: Lib/idlelib/NEWS.txt | 32 ++++++++++++++++++++++++++++++++ 1 files changed, 32 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,6 +1,38 @@ What's New in IDLE 2.7.9? ========================= +*Release data: 2014-12-07* (projected) + +- Issue #16893: Update Idle doc chapter to match current Idle and add new + information. + +- Issue #3068: Add Idle extension configuration dialog to Options menu. + Changes are written to HOME/.idlerc/config-extensions.cfg. + Original patch by Tal Einat. + +- Issue #16233: A module browser (File : Class Browser, Alt+C) requires a + editor window with a filename. When Class Browser is requested otherwise, + from a shell, output window, or 'Untitled' editor, Idle no longer displays + an error box. It now pops up an Open Module box (Alt+M). If a valid name + is entered and a module is opened, a corresponding browser is also opened. + +- Issue #4832: Save As to type Python files automatically adds .py to the + name you enter (even if your system does not display it). Some systems + automatically add .txt when type is Text files. + +- Issue #21986: Code objects are not normally pickled by the pickle module. + To match this, they are no longer pickled when running under Idle. + +- Issue #22221: IDLE now ignores the source encoding declaration on the second + line if the first line contains anything except a comment. + +- Issue #17390: Adjust Editor window title; remove 'Python', + move version to end. + +- Issue #14105: Idle debugger breakpoints no longer disappear + when inseting or deleting lines. + + What's New in IDLE 2.7.8? ========================= -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:51:28 2014 From: python-checkins at python.org (terry.reedy) Date: Sat, 06 Dec 2014 01:51:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Update_idlelib/NEWS=2Etxt=2E?= Message-ID: <20141206015128.76712.98651@psf.io> https://hg.python.org/cpython/rev/37e70e68a2a1 changeset: 93759:37e70e68a2a1 parent: 93756:6f002c4741e2 parent: 93758:3a770955d2bd user: Terry Jan Reedy date: Fri Dec 05 20:51:08 2014 -0500 summary: Update idlelib/NEWS.txt. files: Lib/idlelib/NEWS.txt | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,6 +1,26 @@ What's New in IDLE 3.5.0? ========================= +- Issue #16893: Update Idle doc chapter to match current Idle and add new + information. + +- Issue #3068: Add Idle extension configuration dialog to Options menu. + Changes are written to HOME/.idlerc/config-extensions.cfg. + Original patch by Tal Einat. + +- Issue #16233: A module browser (File : Class Browser, Alt+C) requires a + editor window with a filename. When Class Browser is requested otherwise, + from a shell, output window, or 'Untitled' editor, Idle no longer displays + an error box. It now pops up an Open Module box (Alt+M). If a valid name + is entered and a module is opened, a corresponding browser is also opened. + +- Issue #4832: Save As to type Python files automatically adds .py to the + name you enter (even if your system does not display it). Some systems + automatically add .txt when type is Text files. + +- Issue #21986: Code objects are not normally pickled by the pickle module. + To match this, they are no longer pickled when running under Idle. + - Issue #17390: Adjust Editor window title; remove 'Python', move version to end. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 02:51:28 2014 From: python-checkins at python.org (terry.reedy) Date: Sat, 06 Dec 2014 01:51:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Update_idlelib?= =?utf-8?q?/NEWS=2Etxt=2E?= Message-ID: <20141206015127.76720.87055@psf.io> https://hg.python.org/cpython/rev/3a770955d2bd changeset: 93758:3a770955d2bd branch: 3.4 parent: 93755:6b83e21c8679 user: Terry Jan Reedy date: Fri Dec 05 20:49:32 2014 -0500 summary: Update idlelib/NEWS.txt. files: Lib/idlelib/NEWS.txt | 24 ++++++++++++++++++++++++ 1 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,3 +1,27 @@ +What's New in Idle 3.4.3? +========================= + +- Issue #16893: Update Idle doc chapter to match current Idle and add new + information. + +- Issue #3068: Add Idle extension configuration dialog to Options menu. + Changes are written to HOME/.idlerc/config-extensions.cfg. + Original patch by Tal Einat. + +- Issue #16233: A module browser (File : Class Browser, Alt+C) requires a + editor window with a filename. When Class Browser is requested otherwise, + from a shell, output window, or 'Untitled' editor, Idle no longer displays + an error box. It now pops up an Open Module box (Alt+M). If a valid name + is entered and a module is opened, a corresponding browser is also opened. + +- Issue #4832: Save As to type Python files automatically adds .py to the + name you enter (even if your system does not display it). Some systems + automatically add .txt when type is Text files. + +- Issue #21986: Code objects are not normally pickled by the pickle module. + To match this, they are no longer pickled when running under Idle. + + What's New in IDLE 3.4.2? ========================= -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 03:10:02 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 02:10:02 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_merge_2=2E7=2E9_release_branch?= Message-ID: <20141206021002.76720.15971@psf.io> https://hg.python.org/cpython/rev/b9565af9cb0d changeset: 93761:b9565af9cb0d branch: 2.7 parent: 93757:14ab927a10f6 parent: 93760:aff3a5f813b5 user: Benjamin Peterson date: Fri Dec 05 21:09:58 2014 -0500 summary: merge 2.7.9 release branch files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 03:10:02 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 02:10:02 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_sync_idle_news?= =?utf-8?q?_from_2=2E7_branch?= Message-ID: <20141206021001.73392.65668@psf.io> https://hg.python.org/cpython/rev/aff3a5f813b5 changeset: 93760:aff3a5f813b5 branch: 2.7 parent: 93748:d50096708b2d user: Benjamin Peterson date: Fri Dec 05 21:09:47 2014 -0500 summary: sync idle news from 2.7 branch files: Lib/idlelib/NEWS.txt | 32 ++++++++++++++++++++++++++++++++ 1 files changed, 32 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,6 +1,38 @@ What's New in IDLE 2.7.9? ========================= +*Release data: 2014-12-07* (projected) + +- Issue #16893: Update Idle doc chapter to match current Idle and add new + information. + +- Issue #3068: Add Idle extension configuration dialog to Options menu. + Changes are written to HOME/.idlerc/config-extensions.cfg. + Original patch by Tal Einat. + +- Issue #16233: A module browser (File : Class Browser, Alt+C) requires a + editor window with a filename. When Class Browser is requested otherwise, + from a shell, output window, or 'Untitled' editor, Idle no longer displays + an error box. It now pops up an Open Module box (Alt+M). If a valid name + is entered and a module is opened, a corresponding browser is also opened. + +- Issue #4832: Save As to type Python files automatically adds .py to the + name you enter (even if your system does not display it). Some systems + automatically add .txt when type is Text files. + +- Issue #21986: Code objects are not normally pickled by the pickle module. + To match this, they are no longer pickled when running under Idle. + +- Issue #22221: IDLE now ignores the source encoding declaration on the second + line if the first line contains anything except a comment. + +- Issue #17390: Adjust Editor window title; remove 'Python', + move version to end. + +- Issue #14105: Idle debugger breakpoints no longer disappear + when inseting or deleting lines. + + What's New in IDLE 2.7.8? ========================= -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 04:11:46 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 03:11:46 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_allow_ssl_modu?= =?utf-8?q?le_to_compile_if_openssl_doesn=27t_support_SSL_3_=28closes_=232?= =?utf-8?q?2935=29?= Message-ID: <20141206031143.73394.20054@psf.io> https://hg.python.org/cpython/rev/49d267a58cc2 changeset: 93762:49d267a58cc2 branch: 2.7 parent: 93760:aff3a5f813b5 user: Benjamin Peterson date: Fri Dec 05 21:59:35 2014 -0500 summary: allow ssl module to compile if openssl doesn't support SSL 3 (closes #22935) Patch by Kurt Roeckx. files: Lib/test/test_ftplib.py | 2 +- Lib/test/test_ssl.py | 23 ++++++++++++++++------- Misc/NEWS | 6 ++++++ Modules/_ssl.c | 4 ++++ 4 files changed, 27 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -689,7 +689,7 @@ def test_auth_ssl(self): try: - self.client.ssl_version = ssl.PROTOCOL_SSLv3 + self.client.ssl_version = ssl.PROTOCOL_SSLv23 self.client.auth() self.assertRaises(ValueError, self.client.auth) finally: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2219,20 +2219,24 @@ sys.stdout.write( " SSL2 client to SSL23 server test unexpectedly failed:\n %s\n" % str(x)) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3') + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3') try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1') - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_OPTIONAL) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_OPTIONAL) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, ssl.CERT_OPTIONAL) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_OPTIONAL) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_REQUIRED) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_REQUIRED) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, ssl.CERT_REQUIRED) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED) # Server with specific SSL options - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, False, + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, False, server_options=ssl.OP_NO_SSLv3) # Will choose TLSv1 try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, @@ -2242,6 +2246,8 @@ @skip_if_broken_ubuntu_ssl + @unittest.skipUnless(hasattr(ssl, 'PROTOCOL_SSLv3'), + "OpenSSL is compiled without SSLv3 support") def test_protocol_sslv3(self): """Connecting to an SSLv3 server with various client options""" if support.verbose: @@ -2269,7 +2275,8 @@ try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1) @@ -2284,7 +2291,8 @@ try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1') if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1_1) @@ -2306,7 +2314,8 @@ client_options=ssl.OP_NO_SSLv3|ssl.OP_NO_SSLv2,) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1_2) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,12 @@ - Issue #22960: Add a context argument to xmlrpclib.ServerProxy. +Build +----- + +- Issue #22935: Allow the ssl module to be compiled if openssl doesn't support + SSL 3. + What's New in Python 2.7.9 release candidate 1? =============================================== diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -1997,8 +1997,10 @@ else if (proto_version == PY_SSL_VERSION_TLS1_2) ctx = SSL_CTX_new(TLSv1_2_method()); #endif +#ifndef OPENSSL_NO_SSL3 else if (proto_version == PY_SSL_VERSION_SSL3) ctx = SSL_CTX_new(SSLv3_method()); +#endif #ifndef OPENSSL_NO_SSL2 else if (proto_version == PY_SSL_VERSION_SSL2) ctx = SSL_CTX_new(SSLv2_method()); @@ -4023,8 +4025,10 @@ PyModule_AddIntConstant(m, "PROTOCOL_SSLv2", PY_SSL_VERSION_SSL2); #endif +#ifndef OPENSSL_NO_SSL3 PyModule_AddIntConstant(m, "PROTOCOL_SSLv3", PY_SSL_VERSION_SSL3); +#endif PyModule_AddIntConstant(m, "PROTOCOL_SSLv23", PY_SSL_VERSION_SSL23); PyModule_AddIntConstant(m, "PROTOCOL_TLSv1", -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 04:11:46 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 03:11:46 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjI5MzUp?= Message-ID: <20141206031144.76702.58496@psf.io> https://hg.python.org/cpython/rev/fbf3747e721c changeset: 93765:fbf3747e721c parent: 93759:37e70e68a2a1 parent: 93764:4077e0cd8d48 user: Benjamin Peterson date: Fri Dec 05 22:11:33 2014 -0500 summary: merge 3.4 (#22935) files: Lib/test/test_ftplib.py | 2 +- Lib/test/test_ssl.py | 23 ++++++++++++++++------- Misc/NEWS | 3 +++ Modules/_ssl.c | 4 ++++ 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -889,7 +889,7 @@ def test_auth_ssl(self): try: - self.client.ssl_version = ssl.PROTOCOL_SSLv3 + self.client.ssl_version = ssl.PROTOCOL_SSLv23 self.client.auth() self.assertRaises(ValueError, self.client.auth) finally: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2410,20 +2410,24 @@ sys.stdout.write( " SSL2 client to SSL23 server test unexpectedly failed:\n %s\n" % str(x)) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3') + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3') try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1') - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_OPTIONAL) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_OPTIONAL) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, ssl.CERT_OPTIONAL) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_OPTIONAL) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_REQUIRED) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_REQUIRED) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, ssl.CERT_REQUIRED) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED) # Server with specific SSL options - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, False, + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, False, server_options=ssl.OP_NO_SSLv3) # Will choose TLSv1 try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, @@ -2433,6 +2437,8 @@ @skip_if_broken_ubuntu_ssl + @unittest.skipUnless(hasattr(ssl, 'PROTOCOL_SSLv3'), + "OpenSSL is compiled without SSLv3 support") def test_protocol_sslv3(self): """Connecting to an SSLv3 server with various client options""" if support.verbose: @@ -2460,7 +2466,8 @@ try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1) @@ -2475,7 +2482,8 @@ try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1') if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1_1) @@ -2497,7 +2505,8 @@ client_options=ssl.OP_NO_SSLv3|ssl.OP_NO_SSLv2,) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1_2) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1269,6 +1269,9 @@ Build ----- +- Issue #22935: Allow the ssl module to be compiled if openssl doesn't support + SSL 3. + - Issue #22592: Drop support of the Borland C compiler to build Python. The distutils module still supports it to build extensions. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2132,8 +2132,10 @@ else if (proto_version == PY_SSL_VERSION_TLS1_2) ctx = SSL_CTX_new(TLSv1_2_method()); #endif +#ifndef OPENSSL_NO_SSL3 else if (proto_version == PY_SSL_VERSION_SSL3) ctx = SSL_CTX_new(SSLv3_method()); +#endif #ifndef OPENSSL_NO_SSL2 else if (proto_version == PY_SSL_VERSION_SSL2) ctx = SSL_CTX_new(SSLv2_method()); @@ -4454,8 +4456,10 @@ PyModule_AddIntConstant(m, "PROTOCOL_SSLv2", PY_SSL_VERSION_SSL2); #endif +#ifndef OPENSSL_NO_SSL3 PyModule_AddIntConstant(m, "PROTOCOL_SSLv3", PY_SSL_VERSION_SSL3); +#endif PyModule_AddIntConstant(m, "PROTOCOL_SSLv23", PY_SSL_VERSION_SSL23); PyModule_AddIntConstant(m, "PROTOCOL_TLSv1", -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 04:11:46 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 03:11:46 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_merge_2=2E7=2E9_release_branch?= Message-ID: <20141206031144.21270.92799@psf.io> https://hg.python.org/cpython/rev/a8e6ea473db2 changeset: 93763:a8e6ea473db2 branch: 2.7 parent: 93761:b9565af9cb0d parent: 93762:49d267a58cc2 user: Benjamin Peterson date: Fri Dec 05 22:02:33 2014 -0500 summary: merge 2.7.9 release branch files: Lib/test/test_ftplib.py | 2 +- Lib/test/test_ssl.py | 23 ++++++++++++++++------- Misc/NEWS | 16 +++++++++++----- Modules/_ssl.c | 4 ++++ 4 files changed, 32 insertions(+), 13 deletions(-) diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -689,7 +689,7 @@ def test_auth_ssl(self): try: - self.client.ssl_version = ssl.PROTOCOL_SSLv3 + self.client.ssl_version = ssl.PROTOCOL_SSLv23 self.client.auth() self.assertRaises(ValueError, self.client.auth) finally: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2219,20 +2219,24 @@ sys.stdout.write( " SSL2 client to SSL23 server test unexpectedly failed:\n %s\n" % str(x)) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3') + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3') try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1') - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_OPTIONAL) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_OPTIONAL) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, ssl.CERT_OPTIONAL) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_OPTIONAL) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_REQUIRED) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, 'SSLv3', ssl.CERT_REQUIRED) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, ssl.CERT_REQUIRED) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED) # Server with specific SSL options - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, False, + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, False, server_options=ssl.OP_NO_SSLv3) # Will choose TLSv1 try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, @@ -2242,6 +2246,8 @@ @skip_if_broken_ubuntu_ssl + @unittest.skipUnless(hasattr(ssl, 'PROTOCOL_SSLv3'), + "OpenSSL is compiled without SSLv3 support") def test_protocol_sslv3(self): """Connecting to an SSLv3 server with various client options""" if support.verbose: @@ -2269,7 +2275,8 @@ try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1, 'TLSv1', ssl.CERT_REQUIRED) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1) @@ -2284,7 +2291,8 @@ try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1') if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1_1) @@ -2306,7 +2314,8 @@ client_options=ssl.OP_NO_SSLv3|ssl.OP_NO_SSLv2,) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1_2) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -43,6 +43,12 @@ - Issue #22943: bsddb tests are locale independend now. +IDLE +---- + +- Issue #16893: Update Idle doc chapter to match current Idle and add new + information. + What's New in Python 2.7.9? =========================== @@ -63,11 +69,11 @@ - Issue #22960: Add a context argument to xmlrpclib.ServerProxy. -IDLE ----- - -- Issue #16893: Update Idle doc chapter to match current Idle and add new - information. +Build +----- + +- Issue #22935: Allow the ssl module to be compiled if openssl doesn't support + SSL 3. What's New in Python 2.7.9 release candidate 1? diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -1997,8 +1997,10 @@ else if (proto_version == PY_SSL_VERSION_TLS1_2) ctx = SSL_CTX_new(TLSv1_2_method()); #endif +#ifndef OPENSSL_NO_SSL3 else if (proto_version == PY_SSL_VERSION_SSL3) ctx = SSL_CTX_new(SSLv3_method()); +#endif #ifndef OPENSSL_NO_SSL2 else if (proto_version == PY_SSL_VERSION_SSL2) ctx = SSL_CTX_new(SSLv2_method()); @@ -4023,8 +4025,10 @@ PyModule_AddIntConstant(m, "PROTOCOL_SSLv2", PY_SSL_VERSION_SSL2); #endif +#ifndef OPENSSL_NO_SSL3 PyModule_AddIntConstant(m, "PROTOCOL_SSLv3", PY_SSL_VERSION_SSL3); +#endif PyModule_AddIntConstant(m, "PROTOCOL_SSLv23", PY_SSL_VERSION_SSL23); PyModule_AddIntConstant(m, "PROTOCOL_TLSv1", -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 04:11:46 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 03:11:46 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_allow_ssl_modu?= =?utf-8?q?le_to_compile_if_openssl_doesn=27t_support_SSL_3_=28closes_=232?= =?utf-8?q?2935=29?= Message-ID: <20141206031144.76708.97955@psf.io> https://hg.python.org/cpython/rev/4077e0cd8d48 changeset: 93764:4077e0cd8d48 branch: 3.4 parent: 93758:3a770955d2bd user: Benjamin Peterson date: Fri Dec 05 21:59:35 2014 -0500 summary: allow ssl module to compile if openssl doesn't support SSL 3 (closes #22935) Patch by Kurt Roeckx. files: Lib/test/test_ftplib.py | 2 +- Lib/test/test_ssl.py | 23 ++++++++++++++++------- Misc/NEWS | 3 +++ Modules/_ssl.c | 4 ++++ 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -889,7 +889,7 @@ def test_auth_ssl(self): try: - self.client.ssl_version = ssl.PROTOCOL_SSLv3 + self.client.ssl_version = ssl.PROTOCOL_SSLv23 self.client.auth() self.assertRaises(ValueError, self.client.auth) finally: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2168,20 +2168,24 @@ sys.stdout.write( " SSL2 client to SSL23 server test unexpectedly failed:\n %s\n" % str(x)) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, True) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, True) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, True) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, True, ssl.CERT_OPTIONAL) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, True, ssl.CERT_OPTIONAL) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, ssl.CERT_OPTIONAL) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, True, ssl.CERT_OPTIONAL) - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, True, ssl.CERT_REQUIRED) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, True, ssl.CERT_REQUIRED) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, ssl.CERT_REQUIRED) try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, True, ssl.CERT_REQUIRED) # Server with specific SSL options - try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, False, + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv3, False, server_options=ssl.OP_NO_SSLv3) # Will choose TLSv1 try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv23, True, @@ -2191,6 +2195,8 @@ @skip_if_broken_ubuntu_ssl + @unittest.skipUnless(hasattr(ssl, 'PROTOCOL_SSLv3'), + "OpenSSL is compiled without SSLv3 support") def test_protocol_sslv3(self): """Connecting to an SSLv3 server with various client options""" if support.verbose: @@ -2218,7 +2224,8 @@ try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1, True, ssl.CERT_REQUIRED) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1) @@ -2233,7 +2240,8 @@ try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLSv1_1, True) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1_1) @@ -2255,7 +2263,8 @@ client_options=ssl.OP_NO_SSLv3|ssl.OP_NO_SSLv2,) if hasattr(ssl, 'PROTOCOL_SSLv2'): try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv2, False) - try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False) + if hasattr(ssl, 'PROTOCOL_SSLv3'): + try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv3, False) try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_SSLv23, False, client_options=ssl.OP_NO_TLSv1_2) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -215,6 +215,9 @@ Build ----- +- Issue #22935: Allow the ssl module to be compiled if openssl doesn't support + SSL 3. + - Issue #16537: Check whether self.extensions is empty in setup.py. Patch by Jonathan Hosmer. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2016,8 +2016,10 @@ else if (proto_version == PY_SSL_VERSION_TLS1_2) ctx = SSL_CTX_new(TLSv1_2_method()); #endif +#ifndef OPENSSL_NO_SSL3 else if (proto_version == PY_SSL_VERSION_SSL3) ctx = SSL_CTX_new(SSLv3_method()); +#endif #ifndef OPENSSL_NO_SSL2 else if (proto_version == PY_SSL_VERSION_SSL2) ctx = SSL_CTX_new(SSLv2_method()); @@ -4065,8 +4067,10 @@ PyModule_AddIntConstant(m, "PROTOCOL_SSLv2", PY_SSL_VERSION_SSL2); #endif +#ifndef OPENSSL_NO_SSL3 PyModule_AddIntConstant(m, "PROTOCOL_SSLv3", PY_SSL_VERSION_SSL3); +#endif PyModule_AddIntConstant(m, "PROTOCOL_SSLv23", PY_SSL_VERSION_SSL23); PyModule_AddIntConstant(m, "PROTOCOL_TLSv1", -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Sat Dec 6 10:34:09 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 06 Dec 2014 10:34:09 +0100 Subject: [Python-checkins] Daily reference leaks (37e70e68a2a1): sum=3 Message-ID: results for 37e70e68a2a1 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogQRGv_4', '-x'] From python-checkins at python.org Sat Dec 6 15:32:20 2014 From: python-checkins at python.org (chris.angelico) Date: Sat, 06 Dec 2014 14:32:20 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Small_grammatical_fixes_court?= =?utf-8?q?esy_of_francismb=40email=2Ede?= Message-ID: <20141206143220.74599.87419@psf.io> https://hg.python.org/peps/rev/9f3f176c7116 changeset: 5634:9f3f176c7116 user: Chris Angelico date: Sun Dec 07 01:30:37 2014 +1100 summary: Small grammatical fixes courtesy of francismb at email.de files: pep-0479.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0479.txt b/pep-0479.txt --- a/pep-0479.txt +++ b/pep-0479.txt @@ -160,11 +160,11 @@ A subtle issue is what will happen if the caller, having caught the ``RuntimeError``, calls the generator object's ``__next__()`` method -again. The answer is that it from this point on it will raise +again. The answer is that from this point on it will raise ``StopIteration`` -- the behavior is the same as when any other exception was raised by the generator. -Another logical consequence of the proposal: if somone uses +Another logical consequence of the proposal: if someone uses ``g.throw(StopIteration)`` to throw a ``StopIteration`` exception into a generator, if the generator doesn't catch it (which it could do using a ``try/except`` around the ``yield``), it will be transformed -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Sat Dec 6 16:45:35 2014 From: python-checkins at python.org (chris.angelico) Date: Sat, 06 Dec 2014 15:45:35 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Add_another_example_of_PEP_47?= =?utf-8?q?9_justification=2C_courtesy_of_Isaac_Schwabacher?= Message-ID: <20141206154534.21276.16746@psf.io> https://hg.python.org/peps/rev/eb54574b1791 changeset: 5635:eb54574b1791 user: Chris Angelico date: Sun Dec 07 02:45:03 2014 +1100 summary: Add another example of PEP 479 justification, courtesy of Isaac Schwabacher files: pep-0479.txt | 19 ++++++++++++++++--- 1 files changed, 16 insertions(+), 3 deletions(-) diff --git a/pep-0479.txt b/pep-0479.txt --- a/pep-0479.txt +++ b/pep-0479.txt @@ -72,9 +72,9 @@ print('commit') def do_it(): - print('Refactored preparations') + print('Refactored initial setup') yield # Body of with-statement is executed here - print('Refactored finalization') + print('Refactored finalization of successful transaction') def gene(): for i in range(2): @@ -92,7 +92,20 @@ current behavior this exception will be swallowed by the context manager; and, worse, the finalization is silently skipped! Similarly problematic behavior occurs when an ``asyncio`` coroutine raises -``StopIteration``, causing it to terminate silently. +``StopIteration``, causing it to terminate silently, or when ``next`` +is used to take the first result from an iterator that unexpectedly +turns out to be empty, for example:: + + # using the same context manager as above + import pathlib + + with transaction(): + print('commit file {}'.format( + # I can never remember what the README extension is + next(pathlib.Path('/some/dir').glob('README*')))) + +In both cases, the refactoring abstraction of ``yield from`` breaks +in the presence of bugs in client code. Additionally, the proposal reduces the difference between list comprehensions and generator expressions, preventing surprises such as -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Sat Dec 6 17:12:25 2014 From: python-checkins at python.org (chris.angelico) Date: Sat, 06 Dec 2014 16:12:25 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Reorder/renumber_footnotes_in?= =?utf-8?q?_PEP_479_and_insert_one_for_the_tracker_issue?= Message-ID: <20141206161224.76714.33838@psf.io> https://hg.python.org/peps/rev/bcf0e3459196 changeset: 5636:bcf0e3459196 user: Chris Angelico date: Sun Dec 07 03:12:17 2014 +1100 summary: Reorder/renumber footnotes in PEP 479 and insert one for the tracker issue files: pep-0479.txt | 46 ++++++++++++++++++++------------------- 1 files changed, 24 insertions(+), 22 deletions(-) diff --git a/pep-0479.txt b/pep-0479.txt --- a/pep-0479.txt +++ b/pep-0479.txt @@ -158,7 +158,7 @@ If a ``StopIteration`` is about to bubble out of a generator frame, it is replaced with ``RuntimeError``, which causes the ``next()`` call (which invoked the generator) to fail, passing that exception out. -From then on it's just like any old exception. [4]_ +From then on it's just like any old exception. [3]_ This affects the third outcome listed above, without altering any other effects. Furthermore, it only affects this outcome when the @@ -194,13 +194,15 @@ proposal. Once the feature becomes standard, the flag may be dropped; code should not inspect generators for it. +A proof-of-concept patch has been created to facilitate testing. [4]_ + Consequences for existing code ============================== This change will affect existing code that depends on ``StopIteration`` bubbling up. The pure Python reference -implementation of ``groupby`` [3]_ currently has comments "Exit on +implementation of ``groupby`` [5]_ currently has comments "Exit on ``StopIteration``" where it is expected that the exception will propagate and then be handled. This will be unusual, but not unknown, and such constructs will fail. Other examples abound, e.g. [6]_, [7]_. @@ -474,7 +476,7 @@ Converting the exception inside next() -------------------------------------- -Mark Shannon suggested [12]_ that the problem could be solved in +Mark Shannon suggested [10]_ that the problem could be solved in ``next()`` rather than at the boundary of generator functions. By having ``next()`` catch ``StopIteration`` and raise instead ``ValueError``, all unexpected ``StopIteration`` bubbling would be @@ -490,7 +492,7 @@ Sub-proposal: decorator to explicitly request current behaviour --------------------------------------------------------------- -Nick Coghlan suggested [13]_ that the situations where the current +Nick Coghlan suggested [11]_ that the situations where the current behaviour is desired could be supported by means of a decorator:: from itertools import allow_implicit_stop @@ -523,18 +525,18 @@ compatibility issues created, the subtlety of the decorator's effect, and the fact that it would encourage the "quick-fix" solution of just slapping the decorator onto all generators instead of properly fixing -the code in question, this sub-proposal has been rejected. [14]_ +the code in question, this sub-proposal has been rejected. [12]_ Criticism ========= Unofficial and apocryphal statistics suggest that this is seldom, if -ever, a problem. [5]_ Code does exist which relies on the current +ever, a problem. [13]_ Code does exist which relies on the current behaviour (e.g. [3]_, [6]_, [7]_), and there is the concern that this would be unnecessary code churn to achieve little or no gain. -Steven D'Aprano started an informal survey on comp.lang.python [10]_; +Steven D'Aprano started an informal survey on comp.lang.python [14]_; at the time of writing only two responses have been received: one was in favor of changing list comprehensions to match generator expressions (!), the other was in favor of this PEP's main proposal. @@ -563,14 +565,14 @@ .. [2] Initial mailing list comment (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html) -.. [3] Pure Python implementation of groupby - (https://docs.python.org/3/library/itertools.html#itertools.groupby) - -.. [4] Proposal by GvR +.. [3] Proposal by GvR (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html) -.. [5] Response by Steven D'Aprano - (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) +.. [4] Tracker issue with Proof-of-Concept patch + (http://bugs.python.org/issue22906) + +.. [5] Pure Python implementation of groupby + (https://docs.python.org/3/library/itertools.html#itertools.groupby) .. [6] Split a sequence or generator using a predicate (http://code.activestate.com/recipes/578416-split-a-sequence-or-generator-using-a-predicate/) @@ -584,21 +586,21 @@ .. [9] Coroutines in asyncio (https://docs.python.org/3/library/asyncio-task.html#coroutines) -.. [10] Thread on comp.lang.python started by Steven D'Aprano - (https://mail.python.org/pipermail/python-list/2014-November/680757.html) - -.. [11] Tracker issue with Proof-of-Concept patch - (http://bugs.python.org/issue22906) - -.. [12] Post from Mark Shannon with alternate proposal +.. [10] Post from Mark Shannon with alternate proposal (https://mail.python.org/pipermail/python-dev/2014-November/137129.html) -.. [13] Idea from Nick Coghlan +.. [11] Idea from Nick Coghlan (https://mail.python.org/pipermail/python-dev/2014-November/137201.html) -.. [14] Rejection by GvR +.. [12] Rejection of above idea by GvR (https://mail.python.org/pipermail/python-dev/2014-November/137243.html) +.. [13] Response by Steven D'Aprano + (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) + +.. [14] Thread on comp.lang.python started by Steven D'Aprano + (https://mail.python.org/pipermail/python-list/2014-November/680757.html) + Copyright ========= -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Sat Dec 6 17:37:27 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 16:37:27 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <20141206163723.76724.34282@psf.io> https://hg.python.org/cpython/rev/628acf7dbc8d changeset: 93769:628acf7dbc8d parent: 93765:fbf3747e721c parent: 93768:8839e2432d11 user: Benjamin Peterson date: Sat Dec 06 11:37:18 2014 -0500 summary: merge 3.4 files: Doc/library/ssl.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -554,8 +554,8 @@ Selects SSL version 2 as the channel encryption protocol. - This protocol is not available if OpenSSL is compiled with OPENSSL_NO_SSL2 - flag. + This protocol is not available if OpenSSL is compiled with the + ``OPENSSL_NO_SSL2`` flag. .. warning:: @@ -565,6 +565,9 @@ Selects SSL version 3 as the channel encryption protocol. + This protocol is not be available if OpenSSL is compiled with the + ``OPENSSL_NO_SSLv3`` flag. + .. warning:: SSL version 3 is insecure. Its use is highly discouraged. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 17:37:28 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 16:37:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_note_that_sslv?= =?utf-8?q?3_may_not_be_available?= Message-ID: <20141206163723.76720.64879@psf.io> https://hg.python.org/cpython/rev/8839e2432d11 changeset: 93768:8839e2432d11 branch: 3.4 parent: 93764:4077e0cd8d48 user: Benjamin Peterson date: Sat Dec 06 11:36:32 2014 -0500 summary: note that sslv3 may not be available files: Doc/library/ssl.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -538,8 +538,8 @@ Selects SSL version 2 as the channel encryption protocol. - This protocol is not available if OpenSSL is compiled with OPENSSL_NO_SSL2 - flag. + This protocol is not available if OpenSSL is compiled with the + ``OPENSSL_NO_SSL2`` flag. .. warning:: @@ -549,6 +549,9 @@ Selects SSL version 3 as the channel encryption protocol. + This protocol is not be available if OpenSSL is compiled with the + ``OPENSSL_NO_SSLv3`` flag. + .. warning:: SSL version 3 is insecure. Its use is highly discouraged. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 17:37:28 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 16:37:28 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_merge_2=2E7=2E9_release_branch?= Message-ID: <20141206163723.21266.60790@psf.io> https://hg.python.org/cpython/rev/3b370dbb85b7 changeset: 93767:3b370dbb85b7 branch: 2.7 parent: 93763:a8e6ea473db2 parent: 93766:f18b37befb6c user: Benjamin Peterson date: Sat Dec 06 11:36:48 2014 -0500 summary: merge 2.7.9 release branch files: Doc/library/ssl.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -519,8 +519,8 @@ Selects SSL version 2 as the channel encryption protocol. - This protocol is not available if OpenSSL is compiled with OPENSSL_NO_SSL2 - flag. + This protocol is not available if OpenSSL is compiled with the + ``OPENSSL_NO_SSL2`` flag. .. warning:: @@ -530,6 +530,9 @@ Selects SSL version 3 as the channel encryption protocol. + This protocol is not be available if OpenSSL is compiled with the + ``OPENSSL_NO_SSLv3`` flag. + .. warning:: SSL version 3 is insecure. Its use is highly discouraged. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 17:37:28 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 06 Dec 2014 16:37:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_note_that_sslv?= =?utf-8?q?3_may_not_be_available?= Message-ID: <20141206163723.74597.76479@psf.io> https://hg.python.org/cpython/rev/f18b37befb6c changeset: 93766:f18b37befb6c branch: 2.7 parent: 93762:49d267a58cc2 user: Benjamin Peterson date: Sat Dec 06 11:36:32 2014 -0500 summary: note that sslv3 may not be available files: Doc/library/ssl.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -520,8 +520,8 @@ Selects SSL version 2 as the channel encryption protocol. - This protocol is not available if OpenSSL is compiled with OPENSSL_NO_SSL2 - flag. + This protocol is not available if OpenSSL is compiled with the + ``OPENSSL_NO_SSL2`` flag. .. warning:: @@ -531,6 +531,9 @@ Selects SSL version 3 as the channel encryption protocol. + This protocol is not be available if OpenSSL is compiled with the + ``OPENSSL_NO_SSLv3`` flag. + .. warning:: SSL version 3 is insecure. Its use is highly discouraged. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sat Dec 6 17:44:08 2014 From: python-checkins at python.org (chris.angelico) Date: Sat, 06 Dec 2014 16:44:08 +0000 Subject: [Python-checkins] =?utf-8?q?test=3A_Test_conflict_resolution_on_p?= =?utf-8?q?ush=2C_just_in_case?= Message-ID: <20141206164408.74607.62455@psf.io> https://hg.python.org/test/rev/c26bf6513876 changeset: 220:c26bf6513876 user: Chris Angelico date: Sun Dec 07 03:44:02 2014 +1100 summary: Test conflict resolution on push, just in case files: Lenalede | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lenalede b/Lenalede --- a/Lenalede +++ b/Lenalede @@ -11,3 +11,5 @@ Lenalede your city of free will, giving zones to citizens meriting? .........:.................:..............................:..................... +Originally from Camembert Canyon Campaign. + -- Repository URL: https://hg.python.org/test From python-checkins at python.org Sun Dec 7 01:29:53 2014 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 07 Dec 2014 00:29:53 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322696=3A_Add_func?= =?utf-8?q?tion_=3Afunc=3A=60sys=2Eis=5Ffinalizing=60_to_know_about_interp?= =?utf-8?q?reter?= Message-ID: <20141207002951.21254.99923@psf.io> https://hg.python.org/cpython/rev/7f3695701724 changeset: 93770:7f3695701724 user: Antoine Pitrou date: Sun Dec 07 01:28:27 2014 +0100 summary: Issue #22696: Add function :func:`sys.is_finalizing` to know about interpreter shutdown. files: Doc/glossary.rst | 13 +++++++++++++ Doc/library/gc.rst | 6 +++--- Doc/library/sys.rst | 8 ++++++++ Doc/library/weakref.rst | 6 +++--- Lib/test/test_sys.py | 21 +++++++++++++++++++++ Misc/NEWS | 3 +++ Python/sysmodule.c | 11 +++++++++++ 7 files changed, 62 insertions(+), 6 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -402,6 +402,19 @@ than compiled ones, though their programs generally also run more slowly. See also :term:`interactive`. + interpreter shutdown + When asked to shut down, the Python interpreter enters a special phase + where it gradually releases all allocated resources, such as modules + and various critical internal structures. It also makes several calls + to the :term:`garbage collector `. This can trigger + the execution of code in user-defined destructors or weakref callbacks. + Code executed during the shutdown phase can encounter various + exceptions as the resources it relies on may not function anymore + (common examples are library modules or the warnings machinery). + + The main reason for interpreter shutdown is that the ``__main__`` module + or the script being run has finished executing. + iterable An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as :class:`list`, :class:`str`, diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -186,7 +186,7 @@ added to this list rather than freed. .. versionchanged:: 3.2 - If this list is non-empty at interpreter shutdown, a + If this list is non-empty at :term:`interpreter shutdown`, a :exc:`ResourceWarning` is emitted, which is silent by default. If :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects are printed. @@ -252,8 +252,8 @@ to the ``garbage`` list. .. versionchanged:: 3.2 - Also print the contents of the :data:`garbage` list at interpreter - shutdown, if it isn't empty. + Also print the contents of the :data:`garbage` list at + :term:`interpreter shutdown`, if it isn't empty. .. data:: DEBUG_SAVEALL diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -718,6 +718,14 @@ value of :func:`intern` around to benefit from it. +.. function:: is_finalizing() + + Return :const:`True` if the Python interpreter is + :term:`shutting down `, :const:`False` otherwise. + + .. versionadded:: 3.5 + + .. data:: last_type last_value last_traceback diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -258,7 +258,7 @@ are called in reverse order of creation. A finalizer will never invoke its callback during the later part of - the interpreter shutdown when module globals are liable to have + the :term:`interpreter shutdown` when module globals are liable to have been replaced by :const:`None`. .. method:: __call__() @@ -527,8 +527,8 @@ Starting with Python 3.4, :meth:`__del__` methods no longer prevent reference cycles from being garbage collected, and module globals are -no longer forced to :const:`None` during interpreter shutdown. So this -code should work without any issues on CPython. +no longer forced to :const:`None` during :term:`interpreter shutdown`. +So this code should work without any issues on CPython. However, handling of :meth:`__del__` methods is notoriously implementation specific, since it depends on internal details of the interpreter's garbage diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -744,6 +744,27 @@ c = sys.getallocatedblocks() self.assertIn(c, range(b - 50, b + 50)) + def test_is_finalizing(self): + self.assertIs(sys.is_finalizing(), False) + # Don't use the atexit module because _Py_Finalizing is only set + # after calling atexit callbacks + code = """if 1: + import sys + + class AtExit: + is_finalizing = sys.is_finalizing + print = print + + def __del__(self): + self.print(self.is_finalizing(), flush=True) + + # Keep a reference in the __main__ module namespace, so the + # AtExit destructor will be called at Python exit + ref = AtExit() + """ + rc, stdout, stderr = assert_python_ok('-c', code) + self.assertEqual(stdout.rstrip(), b'True') + @test.support.cpython_only class SizeofTest(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -194,6 +194,9 @@ Library ------- +- Issue #22696: Add function :func:`sys.is_finalizing` to know about + interpreter shutdown. + - Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode will return. This resolves CVE-2013-1753. diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1121,6 +1121,16 @@ "_clear_type_cache() -> None\n\ Clear the internal type lookup cache."); +static PyObject * +sys_is_finalizing(PyObject* self, PyObject* args) +{ + return PyBool_FromLong(_Py_Finalizing != NULL); +} + +PyDoc_STRVAR(is_finalizing_doc, +"is_finalizing()\n\ +Return True if Python is exiting."); + static PyMethodDef sys_methods[] = { /* Might as well keep this in alphabetic order */ @@ -1167,6 +1177,7 @@ getwindowsversion_doc}, #endif /* MS_WINDOWS */ {"intern", sys_intern, METH_VARARGS, intern_doc}, + {"is_finalizing", sys_is_finalizing, METH_NOARGS, is_finalizing_doc}, #ifdef USE_MALLOPT {"mdebug", sys_mdebug, METH_VARARGS}, #endif -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Sun Dec 7 10:33:07 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 07 Dec 2014 10:33:07 +0100 Subject: [Python-checkins] Daily reference leaks (7f3695701724): sum=3 Message-ID: results for 7f3695701724 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogUd8YE6', '-x'] From python-checkins at python.org Sun Dec 7 19:47:48 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 18:47:48 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogdXNlIGNvbnRleHQn?= =?utf-8?q?s_check=5Fhostname_attribute_rather_than_the_HTTPSHandler?= Message-ID: <20141207184748.27507.12385@psf.io> https://hg.python.org/cpython/rev/d5e15f5067c9 changeset: 93774:d5e15f5067c9 branch: 3.4 parent: 93771:a409a7cd908d user: Benjamin Peterson date: Sun Dec 07 13:46:02 2014 -0500 summary: use context's check_hostname attribute rather than the HTTPSHandler check_hostname parameter files: Lib/urllib/request.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -146,10 +146,10 @@ ) if not _have_ssl: raise ValueError('SSL support not available') - context = ssl._create_stdlib_context(cert_reqs=ssl.CERT_REQUIRED, + context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=cafile, capath=capath) - https_handler = HTTPSHandler(context=context, check_hostname=True) + https_handler = HTTPSHandler(context=context) opener = build_opener(https_handler) elif context: https_handler = HTTPSHandler(context=context) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 19:47:47 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 18:47:47 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_remove_HTTPSCo?= =?utf-8?q?nnection=27s_check=5Fhostname_parameter_=28=2322959=29?= Message-ID: <20141207184747.76724.31446@psf.io> https://hg.python.org/cpython/rev/41021c771510 changeset: 93772:41021c771510 branch: 2.7 parent: 93766:f18b37befb6c user: Benjamin Peterson date: Sun Dec 07 13:41:26 2014 -0500 summary: remove HTTPSConnection's check_hostname parameter (#22959) files: Doc/library/httplib.rst | 11 ++--------- Lib/httplib.py | 16 +--------------- Lib/test/data/README | 2 -- Lib/test/test_httplib.py | 13 +++++-------- Lib/urllib2.py | 9 ++++----- Misc/NEWS | 3 +++ 6 files changed, 15 insertions(+), 39 deletions(-) diff --git a/Doc/library/httplib.rst b/Doc/library/httplib.rst --- a/Doc/library/httplib.rst +++ b/Doc/library/httplib.rst @@ -70,7 +70,7 @@ *source_address* was added. -.. class:: HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout[, source_address, context, check_hostname]]]]]]) +.. class:: HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout[, source_address[, context]]]]]]]) A subclass of :class:`HTTPConnection` that uses SSL for communication with secure servers. Default port is ``443``. If *context* is specified, it must @@ -83,13 +83,6 @@ Please read :ref:`ssl-security` for more information on best practices. - .. note:: - If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` - of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then - by default *host* is matched against the host name(s) allowed by the - server's certificate. If you want to change that behaviour, you can - explicitly set *check_hostname* to False. - .. versionadded:: 2.0 .. versionchanged:: 2.6 @@ -99,7 +92,7 @@ *source_address* was added. .. versionchanged:: 2.7.9 - *context* and *check_hostname* was added. + *context* was added. This class now performs all the necessary certificate and hostname checks by default. To revert to the previous, unverified, behavior diff --git a/Lib/httplib.py b/Lib/httplib.py --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -1187,23 +1187,16 @@ def __init__(self, host, port=None, key_file=None, cert_file=None, strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, - source_address=None, context=None, check_hostname=None): + source_address=None, context=None): HTTPConnection.__init__(self, host, port, strict, timeout, source_address) self.key_file = key_file self.cert_file = cert_file if context is None: context = ssl._create_default_https_context() - will_verify = context.verify_mode != ssl.CERT_NONE - if check_hostname is None: - check_hostname = will_verify - elif check_hostname and not will_verify: - raise ValueError("check_hostname needs a SSL context with " - "either CERT_OPTIONAL or CERT_REQUIRED") if key_file or cert_file: context.load_cert_chain(cert_file, key_file) self._context = context - self._check_hostname = check_hostname def connect(self): "Connect to a host on a given (SSL) port." @@ -1217,13 +1210,6 @@ self.sock = self._context.wrap_socket(self.sock, server_hostname=server_hostname) - if not self._context.check_hostname and self._check_hostname: - try: - ssl.match_hostname(self.sock.getpeercert(), server_hostname) - except Exception: - self.sock.shutdown(socket.SHUT_RDWR) - self.sock.close() - raise __all__.append("HTTPSConnection") diff --git a/Lib/test/data/README b/Lib/test/data/README deleted file mode 100644 --- a/Lib/test/data/README +++ /dev/null @@ -1,2 +0,0 @@ -This empty directory serves as destination for temporary files -created by some tests. diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -616,18 +616,15 @@ server = self.make_server(CERT_fakehostname) context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED + context.check_hostname = True context.load_verify_locations(CERT_fakehostname) h = httplib.HTTPSConnection('localhost', server.port, context=context) with self.assertRaises(ssl.CertificateError): h.request('GET', '/') - # Same with explicit check_hostname=True - h = httplib.HTTPSConnection('localhost', server.port, context=context, - check_hostname=True) - with self.assertRaises(ssl.CertificateError): - h.request('GET', '/') - # With check_hostname=False, the mismatching is ignored - h = httplib.HTTPSConnection('localhost', server.port, context=context, - check_hostname=False) + h.close() + # With context.check_hostname=False, the mismatching is ignored + context.check_hostname = False + h = httplib.HTTPSConnection('localhost', server.port, context=context) h.request('GET', '/nonexistent') resp = h.getresponse() self.assertEqual(resp.status, 404) diff --git a/Lib/urllib2.py b/Lib/urllib2.py --- a/Lib/urllib2.py +++ b/Lib/urllib2.py @@ -139,10 +139,10 @@ ) if not _have_ssl: raise ValueError('SSL support not available') - context = ssl._create_stdlib_context(cert_reqs=ssl.CERT_REQUIRED, + context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, cafile=cafile, capath=capath) - https_handler = HTTPSHandler(context=context, check_hostname=True) + https_handler = HTTPSHandler(context=context) opener = build_opener(https_handler) elif context: https_handler = HTTPSHandler(context=context) @@ -1231,14 +1231,13 @@ if hasattr(httplib, 'HTTPS'): class HTTPSHandler(AbstractHTTPHandler): - def __init__(self, debuglevel=0, context=None, check_hostname=None): + def __init__(self, debuglevel=0, context=None): AbstractHTTPHandler.__init__(self, debuglevel) self._context = context - self._check_hostname = check_hostname def https_open(self, req): return self.do_open(httplib.HTTPSConnection, req, - context=self._context, check_hostname=self._check_hostname) + context=self._context) https_request = AbstractHTTPHandler.do_request_ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Library ------- +- Issue #22959: Remove the *check_hostname* parameter of + httplib.HTTPSConnection. The *context* parameter should be used instead. + - Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode will return. This resolves CVE-2013-1753. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 19:47:47 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 18:47:47 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_HTTPSConnectio?= =?utf-8?q?n=3A_prefer_the_context=27s_check=5Fhostname_attribute_over_the?= Message-ID: <20141207184747.21262.24053@psf.io> https://hg.python.org/cpython/rev/a409a7cd908d changeset: 93771:a409a7cd908d branch: 3.4 parent: 93768:8839e2432d11 user: Benjamin Peterson date: Sun Dec 07 13:18:25 2014 -0500 summary: HTTPSConnection: prefer the context's check_hostname attribute over the constructor parameter (#22959) files: Doc/library/http.client.rst | 11 +++-------- Lib/http/client.py | 4 ++-- Lib/test/test_httplib.py | 14 ++++++++++++++ Misc/NEWS | 3 +++ 4 files changed, 22 insertions(+), 10 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -69,17 +69,12 @@ *key_file* and *cert_file* are deprecated, please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let :func:`ssl.create_default_context` select the system's trusted CA - certificates for you. + certificates for you. The *check_hostname* parameter is also deprecated; the + :attr:`SSLContext.check_hostname` attribute of *context* should be used + instead. Please read :ref:`ssl-security` for more information on best practices. - .. note:: - If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` - of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then - by default *host* is matched against the host name(s) allowed by the - server's certificate. If you want to change that behaviour, you can - explicitly set *check_hostname* to False. - .. versionchanged:: 3.2 *source_address*, *context* and *check_hostname* were added. diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -1210,8 +1210,8 @@ context = ssl._create_default_https_context() will_verify = context.verify_mode != ssl.CERT_NONE if check_hostname is None: - check_hostname = will_verify - elif check_hostname and not will_verify: + check_hostname = context.check_hostname + if check_hostname and not will_verify: raise ValueError("check_hostname needs a SSL context with " "either CERT_OPTIONAL or CERT_REQUIRED") if key_file or cert_file: diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -882,6 +882,7 @@ server = self.make_server(CERT_fakehostname) context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED + context.check_hostname = True context.load_verify_locations(CERT_fakehostname) h = client.HTTPSConnection('localhost', server.port, context=context) with self.assertRaises(ssl.CertificateError): @@ -892,11 +893,24 @@ with self.assertRaises(ssl.CertificateError): h.request('GET', '/') # With check_hostname=False, the mismatching is ignored + context.check_hostname = False h = client.HTTPSConnection('localhost', server.port, context=context, check_hostname=False) h.request('GET', '/nonexistent') resp = h.getresponse() self.assertEqual(resp.status, 404) + # The context's check_hostname setting is used if one isn't passed to + # HTTPSConnection. + context.check_hostname = False + h = client.HTTPSConnection('localhost', server.port, context=context) + h.request('GET', '/nonexistent') + self.assertEqual(h.getresponse().status, 404) + # Passing check_hostname to HTTPSConnection should override the + # context's setting. + h = client.HTTPSConnection('localhost', server.port, context=context, + check_hostname=True) + with self.assertRaises(ssl.CertificateError): + h.request('GET', '/') @unittest.skipIf(not hasattr(client, 'HTTPSConnection'), 'http.client.HTTPSConnection not available') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -39,6 +39,9 @@ Library ------- +- Issue #22959: In the constructor of http.client.HTTPSConnection, prefer the + context's check_hostname attribute over the *check_hostname* parameter. + - Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode will return. This resolves CVE-2013-1753. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 19:47:48 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 18:47:48 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_merge_2=2E7=2E9_release_branch?= Message-ID: <20141207184748.21268.15362@psf.io> https://hg.python.org/cpython/rev/bf7d634ceb1b changeset: 93773:bf7d634ceb1b branch: 2.7 parent: 93767:3b370dbb85b7 parent: 93772:41021c771510 user: Benjamin Peterson date: Sun Dec 07 13:41:52 2014 -0500 summary: merge 2.7.9 release branch files: Doc/library/httplib.rst | 11 ++--------- Lib/httplib.py | 16 +--------------- Lib/test/data/README | 2 -- Lib/test/test_httplib.py | 13 +++++-------- Lib/urllib2.py | 9 ++++----- Misc/NEWS | 3 +++ 6 files changed, 15 insertions(+), 39 deletions(-) diff --git a/Doc/library/httplib.rst b/Doc/library/httplib.rst --- a/Doc/library/httplib.rst +++ b/Doc/library/httplib.rst @@ -70,7 +70,7 @@ *source_address* was added. -.. class:: HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout[, source_address, context, check_hostname]]]]]]) +.. class:: HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout[, source_address[, context]]]]]]]) A subclass of :class:`HTTPConnection` that uses SSL for communication with secure servers. Default port is ``443``. If *context* is specified, it must @@ -83,13 +83,6 @@ Please read :ref:`ssl-security` for more information on best practices. - .. note:: - If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` - of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then - by default *host* is matched against the host name(s) allowed by the - server's certificate. If you want to change that behaviour, you can - explicitly set *check_hostname* to False. - .. versionadded:: 2.0 .. versionchanged:: 2.6 @@ -99,7 +92,7 @@ *source_address* was added. .. versionchanged:: 2.7.9 - *context* and *check_hostname* was added. + *context* was added. This class now performs all the necessary certificate and hostname checks by default. To revert to the previous, unverified, behavior diff --git a/Lib/httplib.py b/Lib/httplib.py --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -1191,23 +1191,16 @@ def __init__(self, host, port=None, key_file=None, cert_file=None, strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, - source_address=None, context=None, check_hostname=None): + source_address=None, context=None): HTTPConnection.__init__(self, host, port, strict, timeout, source_address) self.key_file = key_file self.cert_file = cert_file if context is None: context = ssl._create_default_https_context() - will_verify = context.verify_mode != ssl.CERT_NONE - if check_hostname is None: - check_hostname = will_verify - elif check_hostname and not will_verify: - raise ValueError("check_hostname needs a SSL context with " - "either CERT_OPTIONAL or CERT_REQUIRED") if key_file or cert_file: context.load_cert_chain(cert_file, key_file) self._context = context - self._check_hostname = check_hostname def connect(self): "Connect to a host on a given (SSL) port." @@ -1221,13 +1214,6 @@ self.sock = self._context.wrap_socket(self.sock, server_hostname=server_hostname) - if not self._context.check_hostname and self._check_hostname: - try: - ssl.match_hostname(self.sock.getpeercert(), server_hostname) - except Exception: - self.sock.shutdown(socket.SHUT_RDWR) - self.sock.close() - raise __all__.append("HTTPSConnection") diff --git a/Lib/test/data/README b/Lib/test/data/README deleted file mode 100644 --- a/Lib/test/data/README +++ /dev/null @@ -1,2 +0,0 @@ -This empty directory serves as destination for temporary files -created by some tests. diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -639,18 +639,15 @@ server = self.make_server(CERT_fakehostname) context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED + context.check_hostname = True context.load_verify_locations(CERT_fakehostname) h = httplib.HTTPSConnection('localhost', server.port, context=context) with self.assertRaises(ssl.CertificateError): h.request('GET', '/') - # Same with explicit check_hostname=True - h = httplib.HTTPSConnection('localhost', server.port, context=context, - check_hostname=True) - with self.assertRaises(ssl.CertificateError): - h.request('GET', '/') - # With check_hostname=False, the mismatching is ignored - h = httplib.HTTPSConnection('localhost', server.port, context=context, - check_hostname=False) + h.close() + # With context.check_hostname=False, the mismatching is ignored + context.check_hostname = False + h = httplib.HTTPSConnection('localhost', server.port, context=context) h.request('GET', '/nonexistent') resp = h.getresponse() self.assertEqual(resp.status, 404) diff --git a/Lib/urllib2.py b/Lib/urllib2.py --- a/Lib/urllib2.py +++ b/Lib/urllib2.py @@ -139,10 +139,10 @@ ) if not _have_ssl: raise ValueError('SSL support not available') - context = ssl._create_stdlib_context(cert_reqs=ssl.CERT_REQUIRED, + context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, cafile=cafile, capath=capath) - https_handler = HTTPSHandler(context=context, check_hostname=True) + https_handler = HTTPSHandler(context=context) opener = build_opener(https_handler) elif context: https_handler = HTTPSHandler(context=context) @@ -1231,14 +1231,13 @@ if hasattr(httplib, 'HTTPS'): class HTTPSHandler(AbstractHTTPHandler): - def __init__(self, debuglevel=0, context=None, check_hostname=None): + def __init__(self, debuglevel=0, context=None): AbstractHTTPHandler.__init__(self, debuglevel) self._context = context - self._check_hostname = check_hostname def https_open(self, req): return self.do_open(httplib.HTTPSConnection, req, - context=self._context, check_hostname=self._check_hostname) + context=self._context) https_request = AbstractHTTPHandler.do_request_ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -58,6 +58,9 @@ Library ------- +- Issue #22959: Remove the *check_hostname* parameter of + httplib.HTTPSConnection. The *context* parameter should be used instead. + - Issue #16043: Add a default limit for the amount of data xmlrpclib.gzip_decode will return. This resolves CVE-2013-1753. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 19:47:48 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 18:47:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjI5NTkp?= Message-ID: <20141207184748.74599.80491@psf.io> https://hg.python.org/cpython/rev/ee095a2e2b35 changeset: 93775:ee095a2e2b35 parent: 93770:7f3695701724 parent: 93771:a409a7cd908d user: Benjamin Peterson date: Sun Dec 07 13:47:34 2014 -0500 summary: merge 3.4 (#22959) files: Doc/library/http.client.rst | 11 +++-------- Lib/http/client.py | 4 ++-- Lib/test/test_httplib.py | 14 ++++++++++++++ Misc/NEWS | 3 +++ 4 files changed, 22 insertions(+), 10 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -69,17 +69,12 @@ *key_file* and *cert_file* are deprecated, please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let :func:`ssl.create_default_context` select the system's trusted CA - certificates for you. + certificates for you. The *check_hostname* parameter is also deprecated; the + :attr:`SSLContext.check_hostname` attribute of *context* should be used + instead. Please read :ref:`ssl-security` for more information on best practices. - .. note:: - If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` - of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then - by default *host* is matched against the host name(s) allowed by the - server's certificate. If you want to change that behaviour, you can - explicitly set *check_hostname* to False. - .. versionchanged:: 3.2 *source_address*, *context* and *check_hostname* were added. diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -1274,8 +1274,8 @@ context = ssl._create_default_https_context() will_verify = context.verify_mode != ssl.CERT_NONE if check_hostname is None: - check_hostname = will_verify - elif check_hostname and not will_verify: + check_hostname = context.check_hostname + if check_hostname and not will_verify: raise ValueError("check_hostname needs a SSL context with " "either CERT_OPTIONAL or CERT_REQUIRED") if key_file or cert_file: diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -1113,6 +1113,7 @@ server = self.make_server(CERT_fakehostname) context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED + context.check_hostname = True context.load_verify_locations(CERT_fakehostname) h = client.HTTPSConnection('localhost', server.port, context=context) with self.assertRaises(ssl.CertificateError): @@ -1123,11 +1124,24 @@ with self.assertRaises(ssl.CertificateError): h.request('GET', '/') # With check_hostname=False, the mismatching is ignored + context.check_hostname = False h = client.HTTPSConnection('localhost', server.port, context=context, check_hostname=False) h.request('GET', '/nonexistent') resp = h.getresponse() self.assertEqual(resp.status, 404) + # The context's check_hostname setting is used if one isn't passed to + # HTTPSConnection. + context.check_hostname = False + h = client.HTTPSConnection('localhost', server.port, context=context) + h.request('GET', '/nonexistent') + self.assertEqual(h.getresponse().status, 404) + # Passing check_hostname to HTTPSConnection should override the + # context's setting. + h = client.HTTPSConnection('localhost', server.port, context=context, + check_hostname=True) + with self.assertRaises(ssl.CertificateError): + h.request('GET', '/') @unittest.skipIf(not hasattr(client, 'HTTPSConnection'), 'http.client.HTTPSConnection not available') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -194,6 +194,9 @@ Library ------- +- Issue #22959: In the constructor of http.client.HTTPSConnection, prefer the + context's check_hostname attribute over the *check_hostname* parameter. + - Issue #22696: Add function :func:`sys.is_finalizing` to know about interpreter shutdown. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 19:47:48 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 18:47:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <20141207184748.76710.74973@psf.io> https://hg.python.org/cpython/rev/7644f42e3f50 changeset: 93776:7644f42e3f50 parent: 93775:ee095a2e2b35 parent: 93774:d5e15f5067c9 user: Benjamin Peterson date: Sun Dec 07 13:47:39 2014 -0500 summary: merge 3.4 files: Lib/urllib/request.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -146,10 +146,10 @@ ) if not _have_ssl: raise ValueError('SSL support not available') - context = ssl._create_stdlib_context(cert_reqs=ssl.CERT_REQUIRED, + context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=cafile, capath=capath) - https_handler = HTTPSHandler(context=context, check_hostname=True) + https_handler = HTTPSHandler(context=context) opener = build_opener(https_handler) elif context: https_handler = HTTPSHandler(context=context) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 20:19:28 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 19:19:28 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_restore_test_d?= =?utf-8?q?ata_README?= Message-ID: <20141207191928.73384.67020@psf.io> https://hg.python.org/cpython/rev/86095f76f654 changeset: 93777:86095f76f654 branch: 2.7 parent: 93772:41021c771510 user: Benjamin Peterson date: Sun Dec 07 14:19:15 2014 -0500 summary: restore test data README files: Lib/test/data/README | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/data/README b/Lib/test/data/README new file mode 100644 --- /dev/null +++ b/Lib/test/data/README @@ -0,0 +1,2 @@ +This empty directory serves as destination for temporary files +created by some tests, in particular, the test_codecmaps_* tests. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 20:19:29 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 19:19:29 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_merge_2=2E7=2E9_release_branch?= Message-ID: <20141207191928.27499.66093@psf.io> https://hg.python.org/cpython/rev/6e1a825995e3 changeset: 93778:6e1a825995e3 branch: 2.7 parent: 93773:bf7d634ceb1b parent: 93777:86095f76f654 user: Benjamin Peterson date: Sun Dec 07 14:19:24 2014 -0500 summary: merge 2.7.9 release branch files: Lib/test/data/README | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/data/README b/Lib/test/data/README new file mode 100644 --- /dev/null +++ b/Lib/test/data/README @@ -0,0 +1,2 @@ +This empty directory serves as destination for temporary files +created by some tests, in particular, the test_codecmaps_* tests. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 20:26:03 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 19:26:03 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_remove_mention?= =?utf-8?q?_of_check=5Fhostname_parameter?= Message-ID: <20141207192603.21278.93379@psf.io> https://hg.python.org/cpython/rev/4fc7dfd5ecbd changeset: 93779:4fc7dfd5ecbd branch: 2.7 parent: 93777:86095f76f654 user: Benjamin Peterson date: Sun Dec 07 14:25:38 2014 -0500 summary: remove mention of check_hostname parameter files: Doc/library/urllib2.rst | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/urllib2.rst b/Doc/library/urllib2.rst --- a/Doc/library/urllib2.rst +++ b/Doc/library/urllib2.rst @@ -292,13 +292,13 @@ A class to handle opening of HTTP URLs. -.. class:: HTTPSHandler([debuglevel[, context[, check_hostname]]]) +.. class:: HTTPSHandler([debuglevel[, context]]) - A class to handle opening of HTTPS URLs. *context* and *check_hostname* have - the same meaning as for :class:`httplib.HTTPSConnection`. + A class to handle opening of HTTPS URLs. *context* has the same meaning as + for :class:`httplib.HTTPSConnection`. .. versionchanged:: 2.7.9 - *context* and *check_hostname* were added. + *context* added. .. class:: FileHandler() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 20:26:04 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 19:26:04 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_2=2E7=2E9_release_branch?= Message-ID: <20141207192603.21258.3396@psf.io> https://hg.python.org/cpython/rev/ce72e3c45884 changeset: 93780:ce72e3c45884 branch: 2.7 parent: 93778:6e1a825995e3 parent: 93779:4fc7dfd5ecbd user: Benjamin Peterson date: Sun Dec 07 14:25:59 2014 -0500 summary: Merge 2.7.9 release branch files: Doc/library/urllib2.rst | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/urllib2.rst b/Doc/library/urllib2.rst --- a/Doc/library/urllib2.rst +++ b/Doc/library/urllib2.rst @@ -292,13 +292,13 @@ A class to handle opening of HTTP URLs. -.. class:: HTTPSHandler([debuglevel[, context[, check_hostname]]]) +.. class:: HTTPSHandler([debuglevel[, context]]) - A class to handle opening of HTTPS URLs. *context* and *check_hostname* have - the same meaning as for :class:`httplib.HTTPSConnection`. + A class to handle opening of HTTPS URLs. *context* has the same meaning as + for :class:`httplib.HTTPSConnection`. .. versionchanged:: 2.7.9 - *context* and *check_hostname* were added. + *context* added. .. class:: FileHandler() -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 23:26:49 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 22:26:49 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy40?= Message-ID: <20141207222648.27509.18333@psf.io> https://hg.python.org/cpython/rev/dd21f8ef033a changeset: 93782:dd21f8ef033a parent: 93776:7644f42e3f50 parent: 93781:6dfba0357c92 user: Benjamin Peterson date: Sun Dec 07 17:26:43 2014 -0500 summary: merge 3.4 files: Doc/library/http.client.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -70,7 +70,7 @@ :meth:`ssl.SSLContext.load_cert_chain` instead, or let :func:`ssl.create_default_context` select the system's trusted CA certificates for you. The *check_hostname* parameter is also deprecated; the - :attr:`SSLContext.check_hostname` attribute of *context* should be used + :attr:`ssl.SSLContext.check_hostname` attribute of *context* should be used instead. Please read :ref:`ssl-security` for more information on best practices. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Sun Dec 7 23:26:48 2014 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 07 Dec 2014 22:26:48 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_fix_reference_?= =?utf-8?q?by_adding_module_name?= Message-ID: <20141207222648.76710.97196@psf.io> https://hg.python.org/cpython/rev/6dfba0357c92 changeset: 93781:6dfba0357c92 branch: 3.4 parent: 93774:d5e15f5067c9 user: Benjamin Peterson date: Sun Dec 07 17:26:38 2014 -0500 summary: fix reference by adding module name files: Doc/library/http.client.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -70,7 +70,7 @@ :meth:`ssl.SSLContext.load_cert_chain` instead, or let :func:`ssl.create_default_context` select the system's trusted CA certificates for you. The *check_hostname* parameter is also deprecated; the - :attr:`SSLContext.check_hostname` attribute of *context* should be used + :attr:`ssl.SSLContext.check_hostname` attribute of *context* should be used instead. Please read :ref:`ssl-security` for more information on best practices. -- Repository URL: https://hg.python.org/cpython From solipsis at pitrou.net Mon Dec 8 10:34:43 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 08 Dec 2014 10:34:43 +0100 Subject: [Python-checkins] Daily reference leaks (dd21f8ef033a): sum=3 Message-ID: results for dd21f8ef033a on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog1PKrpO', '-x'] From python-checkins at python.org Mon Dec 8 18:22:59 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 17:22:59 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogc2VsZWN0b3JzOiBN?= =?utf-8?q?ake_sure_EpollSelecrtor=2Eselect=28=29_works_when_no_FD_is_regi?= =?utf-8?q?stered=2E?= Message-ID: <20141208172258.76704.70820@psf.io> https://hg.python.org/cpython/rev/b2ee06684b6a changeset: 93783:b2ee06684b6a branch: 3.4 parent: 93781:6dfba0357c92 user: Yury Selivanov date: Mon Dec 08 12:21:58 2014 -0500 summary: selectors: Make sure EpollSelecrtor.select() works when no FD is registered. Closes issue #23009. files: Lib/selectors.py | 7 ++++++- Lib/test/test_selectors.py | 5 +++++ 2 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -418,7 +418,12 @@ # epoll_wait() has a resolution of 1 millisecond, round away # from zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) * 1e-3 - max_ev = len(self._fd_to_key) + + # epoll_wait() expectcs `maxevents` to be greater than zero; + # we want to make sure that `select()` can be called when no + # FD is registered. + max_ev = max(len(self._fd_to_key), 1) + ready = [] try: fd_event_list = self._epoll.poll(timeout, max_ev) diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -319,6 +319,11 @@ self.assertEqual(bufs, [MSG] * NUM_SOCKETS) + def test_empty_select(self): + s = self.SELECTOR() + self.addCleanup(s.close) + self.assertEqual(s.select(timeout=0), []) + def test_timeout(self): s = self.SELECTOR() self.addCleanup(s.close) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 18:22:59 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 17:22:59 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_selectors=3A_Make_sure_EpollSelecrtor=2Eselect=28=29_wor?= =?utf-8?q?ks_when_no_FD_is_registered=2E?= Message-ID: <20141208172259.27519.78074@psf.io> https://hg.python.org/cpython/rev/202995833ef4 changeset: 93784:202995833ef4 parent: 93782:dd21f8ef033a parent: 93783:b2ee06684b6a user: Yury Selivanov date: Mon Dec 08 12:22:33 2014 -0500 summary: selectors: Make sure EpollSelecrtor.select() works when no FD is registered. Closes issue #23009. files: Lib/selectors.py | 7 ++++++- Lib/test/test_selectors.py | 5 +++++ 2 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -418,7 +418,12 @@ # epoll_wait() has a resolution of 1 millisecond, round away # from zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) * 1e-3 - max_ev = len(self._fd_to_key) + + # epoll_wait() expectcs `maxevents` to be greater than zero; + # we want to make sure that `select()` can be called when no + # FD is registered. + max_ev = max(len(self._fd_to_key), 1) + ready = [] try: fd_event_list = self._epoll.poll(timeout, max_ev) diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -316,6 +316,11 @@ self.assertEqual(bufs, [MSG] * NUM_SOCKETS) + def test_empty_select(self): + s = self.SELECTOR() + self.addCleanup(s.close) + self.assertEqual(s.select(timeout=0), []) + def test_timeout(self): s = self.SELECTOR() self.addCleanup(s.close) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 18:30:51 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 17:30:51 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogc2VsZWN0b3JzOiBG?= =?utf-8?q?ix_typo_in_comment=2E?= Message-ID: <20141208173041.74609.42498@psf.io> https://hg.python.org/cpython/rev/6858a2636931 changeset: 93785:6858a2636931 branch: 3.4 parent: 93783:b2ee06684b6a user: Yury Selivanov date: Mon Dec 08 12:30:10 2014 -0500 summary: selectors: Fix typo in comment. files: Lib/selectors.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -419,7 +419,7 @@ # from zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) * 1e-3 - # epoll_wait() expectcs `maxevents` to be greater than zero; + # epoll_wait() expects `maxevents` to be greater than zero; # we want to make sure that `select()` can be called when no # FD is registered. max_ev = max(len(self._fd_to_key), 1) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 18:30:51 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 17:30:51 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_selectors=3A_Fix_typo_in_comment=2E?= Message-ID: <20141208173041.76698.24173@psf.io> https://hg.python.org/cpython/rev/f7faceb6c60a changeset: 93786:f7faceb6c60a parent: 93784:202995833ef4 parent: 93785:6858a2636931 user: Yury Selivanov date: Mon Dec 08 12:30:28 2014 -0500 summary: selectors: Fix typo in comment. files: Lib/selectors.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -419,7 +419,7 @@ # from zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) * 1e-3 - # epoll_wait() expectcs `maxevents` to be greater than zero; + # epoll_wait() expects `maxevents` to be greater than zero; # we want to make sure that `select()` can be called when no # FD is registered. max_ev = max(len(self._fd_to_key), 1) -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 18:38:17 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 17:38:17 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_NEWS=3A_Add_news_entry_for_issue_=2323009=2E?= Message-ID: <20141208173810.74595.60136@psf.io> https://hg.python.org/cpython/rev/d36711410f48 changeset: 93788:d36711410f48 parent: 93786:f7faceb6c60a parent: 93787:8f1be68dfcab user: Yury Selivanov date: Mon Dec 08 12:37:47 2014 -0500 summary: NEWS: Add news entry for issue #23009. files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -194,6 +194,9 @@ Library ------- +- Issue #23009: Make sure selectors.EpollSelecrtor.select() works when no + FD is registered. + - Issue #22959: In the constructor of http.client.HTTPSConnection, prefer the context's check_hostname attribute over the *check_hostname* parameter. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 18:38:17 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 17:38:17 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogTkVXUzogQWRkIG5l?= =?utf-8?q?ws_entry_for_issue_=2323009=2E?= Message-ID: <20141208173810.76722.99906@psf.io> https://hg.python.org/cpython/rev/8f1be68dfcab changeset: 93787:8f1be68dfcab branch: 3.4 parent: 93785:6858a2636931 user: Yury Selivanov date: Mon Dec 08 12:34:06 2014 -0500 summary: NEWS: Add news entry for issue #23009. files: Misc/NEWS | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -178,6 +178,12 @@ - Issue #22448: Improve canceled timer handles cleanup to prevent unbound memory usage. Patch by Joshua Moore-Oliva. +- Issue #22959: In the constructor of http.client.HTTPSConnection, prefer the + context's check_hostname attribute over the *check_hostname* parameter. + +- Issue #23009: Make sure selectors.EpollSelecrtor.select() works when no + FD is registered. + IDLE ---- -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 18:41:17 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 17:41:17 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_null_merge_with_3=2E4=2E?= Message-ID: <20141208174107.74615.96925@psf.io> https://hg.python.org/cpython/rev/1eae51406668 changeset: 93790:1eae51406668 parent: 93788:d36711410f48 parent: 93789:7253417efd1e user: Yury Selivanov date: Mon Dec 08 12:40:53 2014 -0500 summary: null merge with 3.4. files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 18:41:17 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 17:41:17 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogTkVXUzogUmVtb3Zl?= =?utf-8?q?_duplicate_entry?= Message-ID: <20141208174107.21248.56890@psf.io> https://hg.python.org/cpython/rev/7253417efd1e changeset: 93789:7253417efd1e branch: 3.4 parent: 93787:8f1be68dfcab user: Yury Selivanov date: Mon Dec 08 12:39:50 2014 -0500 summary: NEWS: Remove duplicate entry files: Misc/NEWS | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -178,9 +178,6 @@ - Issue #22448: Improve canceled timer handles cleanup to prevent unbound memory usage. Patch by Joshua Moore-Oliva. -- Issue #22959: In the constructor of http.client.HTTPSConnection, prefer the - context's check_hostname attribute over the *check_hostname* parameter. - - Issue #23009: Make sure selectors.EpollSelecrtor.select() works when no FD is registered. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 21:00:29 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 20:00:29 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321740=3A_Support_?= =?utf-8?q?wrapped_callables_in_pydoc=2E_Patch_by_Claudiu_Popa=2E?= Message-ID: <20141208200026.27511.24265@psf.io> https://hg.python.org/cpython/rev/d22ca7496c54 changeset: 93791:d22ca7496c54 user: Yury Selivanov date: Mon Dec 08 15:00:05 2014 -0500 summary: Issue #21740: Support wrapped callables in pydoc. Patch by Claudiu Popa. files: Lib/doctest.py | 3 ++- Lib/test/test_doctest.py | 19 ++++++++++++++++++- Misc/NEWS | 2 ++ 3 files changed, 22 insertions(+), 2 deletions(-) diff --git a/Lib/doctest.py b/Lib/doctest.py --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -985,7 +985,8 @@ for valname, val in obj.__dict__.items(): valname = '%s.%s' % (name, valname) # Recurse to functions & classes. - if ((inspect.isroutine(val) or inspect.isclass(val)) and + if ((inspect.isroutine(inspect.unwrap(val)) + or inspect.isclass(val)) and self._from_module(module, val)): self._find(tests, val, valname, module, source_lines, globs, seen) diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -4,6 +4,7 @@ from test import support import doctest +import functools import os import sys @@ -434,7 +435,7 @@ >>> tests = finder.find(sample_func) >>> print(tests) # doctest: +ELLIPSIS - [] + [] The exact name depends on how test_doctest was invoked, so allow for leading path components. @@ -2364,6 +2365,22 @@ foo \n """ +class Wrapper: + def __init__(self, func): + self.func = func + functools.update_wrapper(self, func) + + def __call__(self, *args, **kwargs): + self.func(*args, **kwargs) + + at Wrapper +def test_look_in_unwrapped(): + """ + Docstrings in wrapped functions must be detected as well. + + >>> 'one other test' + 'one other test' + """ def test_unittest_reportflags(): """Default unittest reporting flags can be set to control reporting diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -194,6 +194,8 @@ Library ------- +- Issue #21740: Support wrapped callables in pydoc. Patch by Claudiu Popa. + - Issue #23009: Make sure selectors.EpollSelecrtor.select() works when no FD is registered. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Mon Dec 8 22:06:12 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 21:06:12 +0000 Subject: [Python-checkins] =?utf-8?q?cpython=3A_inspect=3A_Fix_getsource?= =?utf-8?q?=28=29_to_load_updated_source_of_reloaded_module?= Message-ID: <20141208210602.73400.51268@psf.io> https://hg.python.org/cpython/rev/8e2505d535c8 changeset: 93792:8e2505d535c8 user: Yury Selivanov date: Mon Dec 08 16:05:34 2014 -0500 summary: inspect: Fix getsource() to load updated source of reloaded module Issue #1218234. Initial patch by Berker Peksag. files: Lib/inspect.py | 16 +++++++++---- Lib/test/test_inspect.py | 33 +++++++++++++++++++++++++++- Misc/NEWS | 3 ++ 3 files changed, 46 insertions(+), 6 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -653,11 +653,17 @@ in the file and the line number indexes a line in that list. An OSError is raised if the source code cannot be retrieved.""" - file = getfile(object) - sourcefile = getsourcefile(object) - if not sourcefile and file[:1] + file[-1:] != '<>': - raise OSError('source code not available') - file = sourcefile if sourcefile else file + file = getsourcefile(object) + if file: + # Invalidate cache if needed. + linecache.checkcache(file) + else: + file = getfile(object) + # Allow filenames in form of "" to pass through. + # `doctest` monkeypatches `linecache` module to enable + # inspection, so let `linecache.getlines` to be called. + if not (file.startswith('<') and file.endswith('>')): + raise OSError('source code not available') module = getmodule(object, file) if module: diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -14,6 +14,7 @@ import shutil import sys import types +import textwrap import unicodedata import unittest import unittest.mock @@ -29,6 +30,8 @@ from test import inspect_fodder as mod from test import inspect_fodder2 as mod2 +from test.test_import import _ready_to_import + # Functions tested in this suite: # ismodule, isclass, ismethod, isfunction, istraceback, isframe, iscode, @@ -3262,6 +3265,34 @@ self.assertEqual(err, b'') +class TestReload(unittest.TestCase): + + src_before = textwrap.dedent("""\ +def foo(): + print("Bla") + """) + + src_after = textwrap.dedent("""\ +def foo(): + print("Oh no!") + """) + + def assertInspectEqual(self, path, source): + inspected_src = inspect.getsource(source) + with open(path) as src: + self.assertEqual( + src.read().splitlines(True), + inspected_src.splitlines(True) + ) + + def test_getsource_reload(self): + # see issue 1218234 + with _ready_to_import('reload_bug', self.src_before) as (name, path): + module = importlib.import_module(name) + self.assertInspectEqual(path, module) + with open(path, 'w') as src: + src.write(self.src_after) + self.assertInspectEqual(path, module) def test_main(): @@ -3272,7 +3303,7 @@ TestGetcallargsUnboundMethods, TestGetattrStatic, TestGetGeneratorState, TestNoEOL, TestSignatureObject, TestSignatureBind, TestParameterObject, TestBoundArguments, TestSignaturePrivateHelpers, TestGetClosureVars, - TestUnwrap, TestMain + TestUnwrap, TestMain, TestReload ) if __name__ == "__main__": diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -194,6 +194,9 @@ Library ------- +- Issue #1218234: Fix inspect.getsource() to load updated source of + reloaded module. Initial patch by Berker Peksag. + - Issue #21740: Support wrapped callables in pydoc. Patch by Claudiu Popa. - Issue #23009: Make sure selectors.EpollSelecrtor.select() works when no -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 9 00:01:53 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 23:01:53 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_null_merge_with_3=2E4?= Message-ID: <20141208230143.74611.56730@psf.io> https://hg.python.org/cpython/rev/c426da77695f changeset: 93794:c426da77695f parent: 93792:8e2505d535c8 parent: 93793:e52d8e888df1 user: Yury Selivanov date: Mon Dec 08 18:01:15 2014 -0500 summary: null merge with 3.4 files: -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 9 00:01:53 2014 From: python-checkins at python.org (yury.selivanov) Date: Mon, 08 Dec 2014 23:01:53 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogaW5zcGVjdDogRml4?= =?utf-8?q?_getsource=28=29_to_load_updated_source_of_reloaded_module?= Message-ID: <20141208230143.76720.93322@psf.io> https://hg.python.org/cpython/rev/e52d8e888df1 changeset: 93793:e52d8e888df1 branch: 3.4 parent: 93789:7253417efd1e user: Yury Selivanov date: Mon Dec 08 18:00:25 2014 -0500 summary: inspect: Fix getsource() to load updated source of reloaded module Issue #1218234. Initial patch by Berker Peksag. files: Lib/inspect.py | 16 +++++++++---- Lib/test/test_inspect.py | 33 +++++++++++++++++++++++++++- Misc/NEWS | 3 ++ 3 files changed, 46 insertions(+), 6 deletions(-) diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -652,11 +652,17 @@ in the file and the line number indexes a line in that list. An OSError is raised if the source code cannot be retrieved.""" - file = getfile(object) - sourcefile = getsourcefile(object) - if not sourcefile and file[:1] + file[-1:] != '<>': - raise OSError('source code not available') - file = sourcefile if sourcefile else file + file = getsourcefile(object) + if file: + # Invalidate cache if needed. + linecache.checkcache(file) + else: + file = getfile(object) + # Allow filenames in form of "" to pass through. + # `doctest` monkeypatches `linecache` module to enable + # inspection, so let `linecache.getlines` to be called. + if not (file.startswith('<') and file.endswith('>')): + raise OSError('source code not available') module = getmodule(object, file) if module: diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -12,6 +12,7 @@ import shutil import sys import types +import textwrap import unicodedata import unittest import unittest.mock @@ -27,6 +28,8 @@ from test import inspect_fodder as mod from test import inspect_fodder2 as mod2 +from test.test_import import _ready_to_import + # Functions tested in this suite: # ismodule, isclass, ismethod, isfunction, istraceback, isframe, iscode, @@ -3087,6 +3090,34 @@ self.assertEqual(err, b'') +class TestReload(unittest.TestCase): + + src_before = textwrap.dedent("""\ +def foo(): + print("Bla") + """) + + src_after = textwrap.dedent("""\ +def foo(): + print("Oh no!") + """) + + def assertInspectEqual(self, path, source): + inspected_src = inspect.getsource(source) + with open(path) as src: + self.assertEqual( + src.read().splitlines(True), + inspected_src.splitlines(True) + ) + + def test_getsource_reload(self): + # see issue 1218234 + with _ready_to_import('reload_bug', self.src_before) as (name, path): + module = importlib.import_module(name) + self.assertInspectEqual(path, module) + with open(path, 'w') as src: + src.write(self.src_after) + self.assertInspectEqual(path, module) def test_main(): @@ -3097,7 +3128,7 @@ TestGetcallargsUnboundMethods, TestGetattrStatic, TestGetGeneratorState, TestNoEOL, TestSignatureObject, TestSignatureBind, TestParameterObject, TestBoundArguments, TestSignaturePrivateHelpers, TestGetClosureVars, - TestUnwrap, TestMain + TestUnwrap, TestMain, TestReload ) if __name__ == "__main__": diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -39,6 +39,9 @@ Library ------- +- Issue #1218234: Fix inspect.getsource() to load updated source of + reloaded module. Initial patch by Berker Peksag. + - Issue #22959: In the constructor of http.client.HTTPSConnection, prefer the context's check_hostname attribute over the *check_hostname* parameter. -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Tue Dec 9 04:01:24 2014 From: python-checkins at python.org (guido.van.rossum) Date: Tue, 09 Dec 2014 03:01:24 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_Touch_two_PEPs_with_figures_t?= =?utf-8?q?o_force_re-import_on_www=2Epython=2Eorg=2E?= Message-ID: <20141209030118.678.72621@psf.io> https://hg.python.org/peps/rev/e9cf21f0c0a9 changeset: 5637:e9cf21f0c0a9 user: Guido van Rossum date: Mon Dec 08 19:01:14 2014 -0800 summary: Touch two PEPs with figures to force re-import on www.python.org. files: pep-0458.txt | 1 + pep-0480.txt | 1 - 2 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0458.txt b/pep-0458.txt --- a/pep-0458.txt +++ b/pep-0458.txt @@ -1064,6 +1064,7 @@ .. [25] http://ed25519.cr.yp.to/ .. [26] https://www.python.org/dev/peps/pep-0480/ + Acknowledgements ================ diff --git a/pep-0480.txt b/pep-0480.txt --- a/pep-0480.txt +++ b/pep-0480.txt @@ -883,7 +883,6 @@ Justin Samuel, Tian Tian, Santiago Torres, John Ward, and Yuyu Zheng to develop TUF. - Copyright ========= -- Repository URL: https://hg.python.org/peps From solipsis at pitrou.net Tue Dec 9 10:32:33 2014 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 09 Dec 2014 10:32:33 +0100 Subject: [Python-checkins] Daily reference leaks (c426da77695f): sum=3 Message-ID: results for c426da77695f on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog9IWamA', '-x'] From python-checkins at python.org Tue Dec 9 15:41:59 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 09 Dec 2014 14:41:59 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_add_python-version?= Message-ID: <20141209144154.50880.96162@psf.io> https://hg.python.org/peps/rev/07536f67cd71 changeset: 5638:07536f67cd71 user: Benjamin Peterson date: Tue Dec 09 09:41:53 2014 -0500 summary: add python-version files: pep-0466.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -5,6 +5,7 @@ Author: Nick Coghlan , Status: Final Type: Standards Track +Python-Version: 2.7.9 Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014, 16-Apr-2014 -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Tue Dec 9 17:58:38 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 09 Dec 2014 16:58:38 +0000 Subject: [Python-checkins] =?utf-8?q?test=3A_next_letter?= Message-ID: <20141209165837.672.62823@psf.io> https://hg.python.org/test/rev/95d42e0b43b6 changeset: 221:95d42e0b43b6 user: Benjamin Peterson date: Tue Dec 09 11:58:36 2014 -0500 summary: next letter files: g | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/g b/g --- a/g +++ b/g @@ -1,2 +1,3 @@ gg h +i -- Repository URL: https://hg.python.org/test From python-checkins at python.org Tue Dec 9 23:47:10 2014 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 09 Dec 2014 22:47:10 +0000 Subject: [Python-checkins] =?utf-8?q?peps=3A_put_headers_in_the_right_orde?= =?utf-8?q?r?= Message-ID: <20141209224703.59967.79305@psf.io> https://hg.python.org/peps/rev/18d613ee3d29 changeset: 5639:18d613ee3d29 user: Benjamin Peterson date: Tue Dec 09 17:47:01 2014 -0500 summary: put headers in the right order files: pep-0466.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0466.txt b/pep-0466.txt --- a/pep-0466.txt +++ b/pep-0466.txt @@ -5,9 +5,9 @@ Author: Nick Coghlan , Status: Final Type: Standards Track -Python-Version: 2.7.9 Content-Type: text/x-rst Created: 23-Mar-2014 +Python-Version: 2.7.9 Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014, 16-Apr-2014 Resolution: https://mail.python.org/pipermail/python-dev/2014-April/134163.html -- Repository URL: https://hg.python.org/peps From python-checkins at python.org Wed Dec 10 00:47:41 2014 From: python-checkins at python.org (berker.peksag) Date: Tue, 09 Dec 2014 23:47:41 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzEyNjAy?= =?utf-8?q?=3A_Add_missing_cross-references_to_runpy_and_using/cmdline_doc?= =?utf-8?q?s=2E?= Message-ID: <20141209234740.118688.21548@psf.io> https://hg.python.org/cpython/rev/bd14c4e5ef00 changeset: 93795:bd14c4e5ef00 branch: 3.4 parent: 93793:e52d8e888df1 user: Berker Peksag date: Wed Dec 10 01:47:02 2014 +0200 summary: Issue #12602: Add missing cross-references to runpy and using/cmdline docs. Patch by ?ric Araujo. files: Doc/library/runpy.rst | 13 ++++++++++--- Doc/tutorial/interpreter.rst | 2 ++ Doc/using/cmdline.rst | 12 ++++++++---- 3 files changed, 20 insertions(+), 7 deletions(-) diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -75,6 +75,9 @@ arguments. It is recommended that the :mod:`sys` module be left alone when invoking this function from threaded code. + .. seealso:: + The :option:`-m` option offering equivalent functionality from the + command line. .. versionchanged:: 3.1 Added ability to execute packages by looking for a ``__main__`` submodule. @@ -146,6 +149,10 @@ limitations still apply, use of this function in threaded code should be either serialised with the import lock or delegated to a separate process. + .. seealso:: + :ref:`using-on-interface-options` for equivalent functionality on the + command line (``python path/to/script``). + .. versionadded:: 3.2 .. versionchanged:: 3.4 @@ -156,13 +163,13 @@ .. seealso:: - :pep:`338` - Executing modules as scripts + :pep:`338` -- Executing modules as scripts PEP written and implemented by Nick Coghlan. - :pep:`366` - Main module explicit relative imports + :pep:`366` -- Main module explicit relative imports PEP written and implemented by Nick Coghlan. - :pep:`451` - A ModuleSpec Type for the Import System + :pep:`451` -- A ModuleSpec Type for the Import System PEP written and implemented by Eric Snow :ref:`using-on-general` - CPython command line details diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -63,6 +63,8 @@ and enter interactive mode afterwards. This can be done by passing :option:`-i` before the script. +All command line options are described in :ref:`using-on-general`. + .. _tut-argpassing: diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1,7 +1,7 @@ -.. highlightlang:: none +.. highlightlang:: sh .. ATTENTION: You probably should update Misc/python.man, too, if you modify -.. this file. + this file. .. _using-on-general: @@ -148,6 +148,10 @@ added to the start of :data:`sys.path` and the ``__main__.py`` file in that location is executed as the :mod:`__main__` module. + .. seealso:: + :func:`runpy.run_path` + Equivalent functionality directly available to Python code + If no interface option is given, :option:`-i` is implied, ``sys.argv[0]`` is an empty string (``""``) and the current directory will be added to the @@ -155,11 +159,11 @@ automatically enabled, if available on your platform (see :ref:`rlcompleter-config`). +.. seealso:: :ref:`tut-invoking` + .. versionchanged:: 3.4 Automatic enabling of tab-completion and history editing. -.. seealso:: :ref:`tut-invoking` - Generic options ~~~~~~~~~~~~~~~ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 10 00:47:40 2014 From: python-checkins at python.org (berker.peksag) Date: Tue, 09 Dec 2014 23:47:40 +0000 Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2312602=3A_Add_missing_cross-references_to_runpy_?= =?utf-8?q?and_using/cmdline_docs=2E?= Message-ID: <20141209234740.23836.57413@psf.io> https://hg.python.org/cpython/rev/3a648b3d1694 changeset: 93796:3a648b3d1694 parent: 93794:c426da77695f parent: 93795:bd14c4e5ef00 user: Berker Peksag date: Wed Dec 10 01:47:50 2014 +0200 summary: Issue #12602: Add missing cross-references to runpy and using/cmdline docs. Patch by ?ric Araujo. files: Doc/library/runpy.rst | 13 ++++++++++--- Doc/tutorial/interpreter.rst | 2 ++ Doc/using/cmdline.rst | 12 ++++++++---- 3 files changed, 20 insertions(+), 7 deletions(-) diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -75,6 +75,9 @@ arguments. It is recommended that the :mod:`sys` module be left alone when invoking this function from threaded code. + .. seealso:: + The :option:`-m` option offering equivalent functionality from the + command line. .. versionchanged:: 3.1 Added ability to execute packages by looking for a ``__main__`` submodule. @@ -146,6 +149,10 @@ limitations still apply, use of this function in threaded code should be either serialised with the import lock or delegated to a separate process. + .. seealso:: + :ref:`using-on-interface-options` for equivalent functionality on the + command line (``python path/to/script``). + .. versionadded:: 3.2 .. versionchanged:: 3.4 @@ -156,13 +163,13 @@ .. seealso:: - :pep:`338` - Executing modules as scripts + :pep:`338` -- Executing modules as scripts PEP written and implemented by Nick Coghlan. - :pep:`366` - Main module explicit relative imports + :pep:`366` -- Main module explicit relative imports PEP written and implemented by Nick Coghlan. - :pep:`451` - A ModuleSpec Type for the Import System + :pep:`451` -- A ModuleSpec Type for the Import System PEP written and implemented by Eric Snow :ref:`using-on-general` - CPython command line details diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -63,6 +63,8 @@ and enter interactive mode afterwards. This can be done by passing :option:`-i` before the script. +All command line options are described in :ref:`using-on-general`. + .. _tut-argpassing: diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1,7 +1,7 @@ -.. highlightlang:: none +.. highlightlang:: sh .. ATTENTION: You probably should update Misc/python.man, too, if you modify -.. this file. + this file. .. _using-on-general: @@ -148,6 +148,10 @@ added to the start of :data:`sys.path` and the ``__main__.py`` file in that location is executed as the :mod:`__main__` module. + .. seealso:: + :func:`runpy.run_path` + Equivalent functionality directly available to Python code + If no interface option is given, :option:`-i` is implied, ``sys.argv[0]`` is an empty string (``""``) and the current directory will be added to the @@ -155,11 +159,11 @@ automatically enabled, if available on your platform (see :ref:`rlcompleter-config`). +.. seealso:: :ref:`tut-invoking` + .. versionchanged:: 3.4 Automatic enabling of tab-completion and history editing. -.. seealso:: :ref:`tut-invoking` - Generic options ~~~~~~~~~~~~~~~ -- Repository URL: https://hg.python.org/cpython From python-checkins at python.org Wed Dec 10 01:07:00 2014 From: python-checkins at python.org (berker.peksag) Date: Wed, 10 Dec 2014 00:07:00 +0000 Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyNjAy?= =?utf-8?q?=3A_Add_missing_cross-references_to_runpy_and_using/cmdline_doc?= =?utf-8?q?s=2E?= Message-ID: <20141210000655.88685.75628@psf.io> https://hg.python.org/cpython/rev/078dbecf2e2c changeset: 93797:078dbecf2e2c branch: 2.7 parent: 93780:ce72e3c45884 user: Berker Peksag date: Wed Dec 10 02:07:08 2014 +0200 summary: Issue #12602: Add missing cross-references to runpy and using/cmdline docs. Patch by ?ric Araujo. files: Doc/library/runpy.rst | 11 +++++++++-- Doc/tutorial/interpreter.rst | 2 ++ Doc/using/cmdline.rst | 8 ++++++-- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -72,6 +72,9 @@ arguments. It is recommended that the :mod:`sys` module be left alone when invoking this function from threaded code. + .. seealso:: + The :option:`-m` option offering equivalent functionality from the + command line. .. versionchanged:: 2.7 Added ability to execute packages by looking for a ``__main__`` @@ -134,14 +137,18 @@ limitations still apply, use of this function in threaded code should be either serialised with the import lock or delegated to a separate process. + .. seealso:: + :ref:`using-on-interface-options` for equivalent functionality on the + command line (``python path/to/script``). + .. versionadded:: 2.7 .. seealso:: - :pep:`338` - Executing modules as scripts + :pep:`338` -- Executing modules as scripts PEP written and implemented by Nick Coghlan. - :pep:`366` - Main module explicit relative imports + :pep:`366` -- Main module explicit relative imports PEP written and implemented by Nick Coghlan. :ref:`using-on-general` - CPython command line details diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -62,6 +62,8 @@ and enter interactive mode afterwards. This can be done by passing :option:`-i` before the script. +All command-line options are described in :ref:`using-on-general`. + .. _tut-argpassing: diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1,7 +1,7 @@ -.. highlightlang:: none +.. highlightlang:: sh .. ATTENTION: You probably should update Misc/python.man, too, if you modify -.. this file. + this file. .. _using-on-general: @@ -130,6 +130,10 @@ ``"-"`` and the current directory will be added to the start of :data:`sys.path`. + .. seealso:: + :func:`runpy.run_path` + Equivalent functionality directly available to Python code + .. describe::